[MAPREDUCE-2657] TaskTracker should handle disk failures

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.20.203.0
• Component/s: tasktracker
• Assignee: Unassigned
• Reporter: Bharath Mundlapudi
• Created: 28/Jan/11 21:05
• Updated: 08/Sep/11 21:11

Description

Umbrella for TaskTracker disk failure handling jiras.

Issue Links

• is part of: HADOOP-7123 Hadoop Disk Fail Inplace
• is related to: HADOOP-7551 LocalDirAllocator should incorporate LocalStorage
• relates to: MAPREDUCE-2656 Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.
• relates to: MAPREDUCE-2651 Race condition in Linux Task Controller for job log directory creation

Sub-Tasks

• TaskTracker should handle disk failures at both startup and runtime: Closed
• Distribute TaskTracker userlogs onto multiple disks: Closed
• Add test for TaskTracker disk failure handling (MR-2413): Closed
• Add test for userlogs on multiple disks (MR-2415): Open
• Map Reduce Tasks are continously failing, when one among the several harddisks available on the TaskTracker fails.: Resolved
• If Localizer can't create task log directory, it should fail on the spot: Closed
• Local log dir links in the JT web UI are broken: Closed
• TaskTracker won't start with failed local directory: Resolved
• TaskTracker number of failed disks to tolerate should be configurable: Closed
• MR-2413 improvements: Closed
• The TT should not re-init if it has no good local dirs: Closed
• A single TT disk failure can cause the job to fail: Open
• TT should remove bad local dirs from conf to prevent constant disk checking: Closed
• Add local dir failure info to metrics and the web UI: Closed
• Add TT local dir failure info to the JT web UI: Resolved
• Some LinuxTaskController cleanup: Closed