JT fails to recover the jobs after restart after HADOOP:4372

• Key: HADOOP-5924
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Amar Kamat
• Reporter: Ramya R
• Fix Version/s: 0.20.1

Comments

[Ramya R]

Submitted a job and restarted the JT after sometime. Below is the snapshot of the JT log:

INFO org.apache.hadoop.mapred.JobTracker: Submitting job <jobID> on behalf of user <user> in groups :<group>
INFO org.apache.hadoop.mapred.JobHistory: Recovered job history filename for job <jobID> is <job history file>
INFO org.apache.hadoop.mapred.JobHistory:  <job history file> exists!
INFO org.apache.hadoop.mapred.JobHistory: <job history file> exists!
INFO org.apache.hadoop.mapred.JobQueuesManager: Job submitted to queue default
WARN org.apache.hadoop.fs.FSInputChecker: Problem opening checksum file: file:<logs>history/<job history file>
Ignoring exception: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:180)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:134)
        at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:351)
        at org.apache.hadoop.mapred.JobHistory.parseHistoryFromFS(JobHistory.java:254)
        at org.apache.hadoop.mapred.JobTracker$RecoveryManager.recover(JobTracker.java:1361)
        at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:1850)
        at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3695)
INFO org.apache.hadoop.mapred.JobHistory: Deleting job history file <job history file>
INFO org.apache.hadoop.mapred.JobTracker: Restoration complete
INFO org.apache.hadoop.mapred.JobInitializationPoller: Passing to Initializer Job Id :<jobID> User:<user> Queue : default
INFO org.apache.hadoop.mapred.JobInitializationPoller: Initializing job : <jobID> in Queue default For user : <user>
INFO org.apache.hadoop.mapred.JobInProgress: Initializing <jobID>
INFO org.apache.hadoop.mapred.JobHistory: Nothing to recover for job <jobID>
INFO org.apache.hadoop.mapred.JobInitializationPoller: Job initialization failed:
java.lang.IllegalArgumentException: Can not create a Path from a null string
        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
        at org.apache.hadoop.fs.Path.<init>(Path.java:90)
        at org.apache.hadoop.fs.Path.<init>(Path.java:45)
        at org.apache.hadoop.mapred.JobHistory$JobInfo.getJobHistoryLogLocation(JobHistory.java:577)
        at org.apache.hadoop.mapred.JobHistory$JobInfo.logSubmitted(JobHistory.java:871)
        at org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:405)
        at org.apache.hadoop.mapred.JobInitializationPoller$JobInitializationThread.initializeJobs(JobInitializationPoller.java:143)
        at org.apache.hadoop.mapred.JobInitializationPoller$JobInitializationThread.run(JobInitializationPoller.java:113)
INFO org.apache.hadoop.mapred.JobHistory: Nothing to recover for job <jobID>
INFO org.apache.hadoop.mapred.JobInitializationPoller: Removing killed/completed job from initalized jobs list : <jobID>

The job fails to recover and is marked as failed. This happens for all the jobs(irrespective of map/reduce progress)

[Amar Kamat]

Attaching a patch that fixes the issue. Result of test-patch

[exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Running ant test now.

[Amar Kamat]

ant test passed on my box.

[Devaraj Das]

Minor nit - the check for an empty killList is redundant and can be removed.

[Amar Kamat]

Attaching a patch incorporating Devaraj's comments. Result of test-patch

[exec] +1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 9 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec] 
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.

Note that the patch depends on HADOOP-5908.

Ant tests passed on my box.

[Amar Kamat]

Attaching a patch for 0.20 branch.

[Devaraj Das]

I just committed this. Thanks, Amar!

[Robert Chansler]

Attached an alternate version for 0.20 not to be committed to the branch.