Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
This is caused by a missing block in HDFS. So the block's locations are empty. The following code adds the block to blockToNodes map but not to rackToBlocks map. Later on when generating splits, only blocks in rackToBlocks are removed from blockToNodes map. So blockToNodes map can never become empty therefore causing infinite loop
// add this block to the block --> node locations map blockToNodes.put(oneblock, oneblock.hosts); // add this block to the rack --> block map for (int j = 0; j < oneblock.racks.length; j++) { .. }
For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as the rack location. This avoids the infinite loop later on in getMoreSplits()