[HADOOP-4664] Parallelize job initialization

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.20.0
• Hadoop Flags: Reviewed
• Assignee: Jothi Padmanabhan
• Reporter: Matei Alexandru Zaharia
• Created: 15/Nov/08 04:45
• Updated: 08/Jul/09 16:53
• Resolved: 12/Mar/09 16:59

Description

The job init thread currently initializes one job at a time. However, this is a lengthy and partly IO-bound process because all of the job's block locations need to be resolved through the namenode and a map of them needs to be built. It can take tens of seconds. As a result, the cluster sometimes initializes jobs too slowly for full utilization to be achieved, if there are many small jobs queued up. It would be better to have a pool of threads that initialize multiple jobs in parallel. One thing to be careful of, however, is not causing deadlocks or holding locks for too long in these threads.

Issue Links

• incorporates: HADOOP-5286 DFS client blocked for a long time reading blocks of a file on the JobTracker