Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2185

Infinite loop at creating splits using CombineFileInputFormat

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: job submission
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is caused by a missing block in HDFS. So the block's locations are empty. The following code adds the block to blockToNodes map but not to rackToBlocks map. Later on when generating splits, only blocks in rackToBlocks are removed from blockToNodes map. So blockToNodes map can never become empty therefore causing infinite loop

                // add this block to the block --> node locations map
                blockToNodes.put(oneblock, oneblock.hosts);
      
                // add this block to the rack --> block map
                for (int j = 0; j < oneblock.racks.length; j++) {
                   ..
                }
      
      1. MAPREDUCE-2185.patch
        7 kB
        Ramkumar Vadali

        Activity

        Hide
        Ramkumar Vadali added a comment -

        For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as the rack location. This avoids the infinite loop later on in getMoreSplits()

        Show
        Ramkumar Vadali added a comment - For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as the rack location. This avoids the infinite loop later on in getMoreSplits()
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480153/MAPREDUCE-2185.patch
        against trunk revision 1126591.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480153/MAPREDUCE-2185.patch against trunk revision 1126591. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        +1 code looks good to me.

        Show
        dhruba borthakur added a comment - +1 code looks good to me.
        Hide
        Scott Chen added a comment -

        I have committed this. Thanks Ram for the fix and dhruba for the review.

        Show
        Scott Chen added a comment - I have committed this. Thanks Ram for the fix and dhruba for the review.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/)
        MAPREDUCE-2185. Fix infinite loop at creating splits using CombineFileInputFormat. (Ramkumar Vadali via schen)

        schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807
        Files :

        • /hadoop/mapreduce/trunk/CHANGES.txt
        • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
        • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/ ) MAPREDUCE-2185 . Fix infinite loop at creating splits using CombineFileInputFormat. (Ramkumar Vadali via schen) schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development