Improve the way the job history files are managed during job recovery

• Key: HADOOP-4372
• Type: Improvement
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Amar Kamat
• Reporter: Amar Kamat
• Fix Version/s: 0.21.0

Description

Today we use the .recover technique to handle the job history files when the jobtracker restarts. The comment here proposes a better way to handle the files.

Comments

[Amar Kamat]

Attaching a patch the implements the basic idea. This patch uses a new filename on recovery. Updated the testcase accordingly.

[Amar Kamat]

Result of test-patch

[exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Ant tests passed on my box.

[Devaraj Das]

Could we instead pass a boolean to logSubmitted depending on the restart-count value (0 == new job, and the boolean is true), and then create the recovery file inside the jobHistory if the boolean is false.
The problem with the current patch is that the filename of the history file changes upon restart..

[Amar Kamat]

Attaching a patch that simply uses the boolean passed whether the job is old or new. Testing in progress.

[Amar Kamat]

Changes to src/mapred/mapred-default.conf are not required. Plz ignore that.

[Amar Kamat]

Attaching a patch that optimizes jobhistory for new jobs. Result of test-patch

[exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     -1 tests included.  The patch doesn't appear to include any new or modified tests.
     [exec]                         Please justify why no tests are needed for this patch.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Ant test passed on my box

[Ravi Gummadi]

Patch looks good.
+1

[Devaraj Das]

I just committed this. Thanks, Amar!

[Hudson]

Integrated in Hadoop-trunk #830 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/830/)
. Improves the way history filenames are obtained and manipulated. Contributed by Amar Kamat.

[Devaraj Das]

I committed this to 0.20 branch as well. There have been problems like job submission taking a long time when the number of files in the history folder are too large. This patch introduced an API that would fasten the process of obtaining a history file for a new job (earlier it used to do a scan of the history folder which is costly when the number of files are large).

[Hudson]

Integrated in Hadoop-trunk #863 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/)