[MAPREDUCE-751] Rumen: a tool to extract job characterization data from job tracker logs

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.21.0
• Component/s: tools/rumen
• Hadoop Flags: Reviewed
• Tags: rumen,mumakil,job tracker logs
• Assignee: Dick King
• Reporter: Dick King
• Created: 12/Jul/09 04:04
• Updated: 24/Nov/11 14:34
• Resolved: 28/Aug/09 00:13

Description

We propose a new map/reduce component, rumen, which can be used to process job history logs to produce any or all of the following:

• Retrospective info describing the statistical behavior of the
  amount of time it would have taken to launch a job into a certain
  percentage of the number of mapper slots in the log's cluster, given the
  load over the period covered by the log

• Statistical info as to the runtimes and shuffle times, etc. of
  the tasks and jobs covered by the log

• files describing detailed job trace information, and the
  network topology as inferred from the host locations and rack IDs that
  arise in the job tracker log. In addition to this facility, rumen
  includes readers for this information to return job and detailed task
  information to other tools.

These other tools include a more advanced version of gridmix, and also includes mumak: see blocked issues.

Issue Links

• blocks: MAPREDUCE-728 Mumak: Map-Reduce Simulator
• blocks: MAPREDUCE-776 Gridmix: Trace-based benchmark for Map/Reduce
• relates to: CHUKWA-342 Static Swimlanes Visualization Widget