[MAPREDUCE-2862] Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Kazuki Ohta
• Created: 19/Aug/11 16:55
• Updated: 23/Dec/11 17:37

Description

Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits().

At first, we lost some blocks by mis-operation . Then, one job tried to use these missing blocks. At that time getMoreSplits() goes into the infinite loop.

From our investigation, this List could be an empty array.
> https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363

Then 'for' loop just after that line does nothing, and entry is not removed from 'blockToNodes'.

Finally this line goes into the infinite loop.
> https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348

We're now creating the patch against this problem...