- Overview
- Conference Organizers
- Registration Information
- Registration Discounts
- At a Glance
- Calendar
- Activities
- Technical Sessions
- Workshops
- Posters and Demos
- Birds-of-a-Feather Sessions
- Sponsorship
- Hotel and Travel Information
- Services
- Students
- Questions
- Help Promote!
- For Participants
- Call for Papers
- Past Conferences
sponsors
usenix conference policies
FMEM: A Fine-grained Memory Estimator for MapReduce Jobs
Lijie Xu, Institute of Software, Chinese Academy of Sciences, and University of Chinese Academy of Sciences; Jie Liu and Jun Wei, Institute of Software, Chinese Academy of Sciences
MapReduce is designed as a simple and scalable framework for big data processing. Due to the lack of resource usage models, its implementation Hadoop hands over resource planning and optimizing works to users. But users also find difficulty in specifying right resource-related, especially memory-related, configurations without good knowledge of job’s memory usage. Modeling memory usage is challenging because there are many influencing factors such as framework’s dataflow, user-defined programs, large space of configurations and memory management mechanism of JVM. In order to help both users and the framework to analyze, predict and optimize memory usage, we propose a Fine-grained Memory Estimator for MapReduce jobs called FMEM. FMEM contains a dataflow estimator which can predict the data volume flowing among map/reduce tasks. Based on dataflow and rules of memory utilization learnt from a lot of jobs, FMEM uses a rules-statistics method to estimate fine-grained memory usage in each generation of task’s JVM. Representative benchmarks show that FMEM can predict diverse jobs' memory usage within 20% relative error. Furthermore, FMEM will be promoted to find optimum dataflow and memory related configurations.
Open Access Media
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
author = {Lijie Xu and Jie Liu and Jun Wei},
title = {{FMEM}: A Fine-grained Memory Estimator for {MapReduce} Jobs},
booktitle = {10th International Conference on Autonomic Computing (ICAC 13)},
year = {2013},
isbn = {978-1-931971-02-7},
address = {San Jose, CA},
pages = {65--68},
url = {https://www.usenix.org/conference/icac13/technical-sessions/presentation/xu_lijie},
publisher = {USENIX Association},
month = jun
}
connect with us