[HADOOP-3245] Provide ability to persist running jobs (extend HADOOP-1876)

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.19.0
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: Introduced recovery of jobs when JobTracker restarts. This facility is off by default. Introduced config parameters mapred.jobtracker.restart.recover, mapred.jobtracker.job.history.block.size, and mapred.jobtracker.job.history.buffer.size.
• Assignee: Amar Kamat
• Reporter: Devaraj Das
• Created: 14/Apr/08 08:43
• Updated: 02/May/13 02:29
• Resolved: 16/Sep/08 12:06

Description

This could probably extend the work done in HADOOP-1876. This feature can be applied for things like jobs being able to survive jobtracker restarts.

Issue Links

• blocks: MAPREDUCE-277 Job history counters should be avaible on the UI.
• depends upon: HDFS-53 RawLocalFileSystem.listStatus() should not throw FileNotFound exception if a file goes missing while the list is being computed
• depends upon: MAPREDUCE-7 Reduce progress in the Reducer crosses 1
• depends upon: HADOOP-2403 JobHistory log files contain data that cannot be parsed by org.apache.hadoop.mapred.JobHistory
• depends upon: HADOOP-3970 Counters written to the job history cannot be recovered back
• is blocked by: HADOOP-3590 Null pointer exception in JobTracker when the task tracker is not yet resolved
• is blocked by: HADOOP-3947 TaskTrackers fail to connect back upon a re-init action
• is blocked by: HADOOP-3641 Exception while killing the JobTracker
• is blocked by: HADOOP-4112 Got ArrayOutOfBound exception while analyzing the job history
• is blocked by: HADOOP-3780 JobTracker should synchronously resolve the tasktracker's network location when the tracker registers
• is depended upon by: MAPREDUCE-225 Fault tolerant Hadoop Job Tracker
• is part of: HADOOP-3444 Implementing a Resource Manager (V1) for Hadoop
• is related to: HADOOP-5436 job history directory grows without bound, locks up job tracker on new job submission
• is related to: HADOOP-4220 Job Restart tests take 10 minutes, can time out very easily
• relates to: HADOOP-3771 JobClient.runJob() should not kill the job on IOExceptions