MapReduce and Other Building Blocks for Large-Scale Distributed Systems at Google

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a Map function which processes a key/value pair to generate a set of intermediate key/value pairs, and a Reduce function which merges all intermediate values associated with the same intermediate key. Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The MapReduce run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system. Thousands of MapReduce programs have been implemented, and several thousand MapReduce jobs are executed on Google's clusters every day. In this talk I'll describe the design and implementation of MapReduce and other building blocks for large-scale distributed systems at Google.

Jeff is a Google Fellow in Google's Systems Infrastructure Group. While at Google, he has worked on Google's crawling, indexing, query serving, and advertising systems and has built various pieces of Google's distributed computing infrastructure. Prior to joining Google, he was at DEC/Compaq's Western Research Laboratory, where he worked on profiling tools, microprocessor architecture, and information retrieval. Prior to graduate school, he worked at the World Health Organization's Global Programme on AIDS.