Transparent and Flexible Network Management for Big Data Processing in the Cloud
Authors:
Anupam Das, University of Illinois at Urbana-Champaign; Cristian Lumezanu, Yueping Zhang, Vishal Singh, and Guofei Jiang, NEC Labs; Curtis Yu, University of California, Riverside
Abstract:
We introduce FlowComb, a network management framework that helps Big Data processing applications, such as Hadoop, achieve high utilization and low data processing times. FlowComb predicts application network transfers, sometimes before they start, by using software agents installed on application servers and while remaining completely transparent to the application. A centralized decision engine collects data movement information from agents and schedules upcoming flows on paths such that the network does not become congested. Results on our lab testbed show that FlowComb is able to reduce the time to sort 10GB of randomly generated data by 35% while changing paths for only 6% of the transfers.
USENIX is committed to Open Access to the research presented at our events. Papers and proceedings are freely available to everyone once the event begins. Any video, audio, and/or slides that are posted after the event are also free and open to everyone. Support USENIX and our commitment to Open Access.
BibTeX
@inproceedings {179401,
author = {Anupam Das and Cristian Lumezanu and Yueping Zhang and Vishal Singh and Guofei Jiang and Curtis Yu},
title = {Transparent and Flexible Network Management for Big Data Processing in the Cloud},
booktitle = {5th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 13)},
year = {2013},
address = {San Jose, CA},
url = {https://www.usenix.org/conference/hotcloud13/workshop-program/presentations/das},
publisher = {USENIX Association},
month = jun
}
connect with us