[HADOOP-307] Many small jobs benchmark for MapReduce

• Type: Task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Assignee: Sanjay Dahiya
• Reporter: Sanjay Dahiya
• Created: 18/Jun/06 20:14
• Updated: 08/Jul/09 16:51
• Resolved: 18/Jul/06 11:03

Description

A benchmark that runs many small MapReduce tasks in sequence. A single map reduce implementation is used, it is invoked multiple times with input as the output from previous run. The input to first Map is a TextInputFormat ( a text file with few hundred KBs). Input records are passed to output without much processing. The idea is to benchmark the time taken by initialization of Mapper and Reducer. An initial prototyping on a single machine with 20 MR tasks in sequence took ~47 seconds per task. Looking for suggestions on what else can be included in the benchmark.

Issue Links

• incorporates: HADOOP-434 Use Hadoop scripts to run smallJobsBenchmark to avoid classpath issues.
• is blocked by: HADOOP-460 Small jobs benchmark fails with current Hadoop due to UTF8 -> Text ClassCastException