Using Hadoop for Webscale Computing

Apache Hadoop is an open source implementation of a distributed filesystem and map-reduce programming model combined into one package. Hadoop scales smoothly from tens to thousands of computers. The framework allows engineers to harness the power of these clusters very simply, taking advantage of three major features:

• A reliable, non-hardware-based distributed filesystem: Hadoop DFS runs on any number of nodes, taking advantage of their combined storage to manage replication and recovery from failure.
• A simple, functional programming model: Hadoop Map-Reduce is a parallelized implementation of a very simple programming methodology first popularized by the functional programming group in the 1970s.
• Infrastructure to aid in the automation of job execution: Hadoop automates bringing user code to the data, and it manages parallel execution and handles node failure.

The talk will provide an overview of Apache Hadoop, along with examples of how this infrastructure is being used at Yahoo! and other organizations today.