[HADOOP-4517] unstable dfs when running jobs on 0.18.1

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.18.1
• Fix Version/s: 0.18.2

Environment

hadoop-0.18.1 plus patches HADOOP-4277 HADOOP-4271 HADOOP-4326 HADOOP-4314 HADOOP-3914 HADOOP-4318 HADOOP-4351 HADOOP-4395

• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Christian Kunz
• Created: 24/Oct/08 19:16
• Updated: 08/Jul/09 16:43
• Resolved: 28/Oct/08 00:07

Description

2 attempts of a job using 6000 maps, 1900 reduces

1.st attempt: failed during reduce phase after 22 hours with 31 dead datanodes most of which became unresponsive due to an exception; dfs lost blocks
2nd attempt: failed during map phase after 5 hours with 5 dead datanodes due to exception; dfs lost blocks responsible for job failure.

I will post typical datanode exception and attach thread dump.