[MAPREDUCE-2010] [Rumen] Parallelize TraceBuilder

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.22.0
• Component/s: tools/rumen
• Assignee: Amar Kamat
• Reporter: Amar Kamat
• Created: 13/Aug/10 11:20
• Updated: 13/Jan/11 02:29

Description

Currently, Rumen's TraceBuilder processes jobs in sequential manner and emits them in sorted order (based on job-id). Following are the steps :

1. Read data from input files
2. Parse and analyze the JobHistory data
3. Write the data to the output file

Steps #1 and #2 can be done in parallel. Step #3 can be made sequential (if user needs it) else can also be done in parallel.

I could achieve ~50% speedup by simply parallelizing step#1 and step#2 (i.e output was sorted based on job-id).