We propose a new map/reduce component, rumen, which can be used to process job history logs to produce any or all of the following:
- Retrospective info describing the statistical behavior of the
amount of time it would have taken to launch a job into a certain
percentage of the number of mapper slots in the log's cluster, given the
load over the period covered by the log
- Statistical info as to the runtimes and shuffle times, etc. of
the tasks and jobs covered by the log
- files describing detailed job trace information, and the
network topology as inferred from the host locations and rack IDs that
arise in the job tracker log. In addition to this facility, rumen
includes readers for this information to return job and detailed task
information to other tools.
These other tools include a more advanced version of gridmix, and also includes mumak: see blocked issues.