Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2185

Infinite loop at creating splits using CombineFileInputFormat

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: job submission
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is caused by a missing block in HDFS. So the block's locations are empty. The following code adds the block to blockToNodes map but not to rackToBlocks map. Later on when generating splits, only blocks in rackToBlocks are removed from blockToNodes map. So blockToNodes map can never become empty therefore causing infinite loop

                // add this block to the block --> node locations map
                blockToNodes.put(oneblock, oneblock.hosts);
      
                // add this block to the rack --> block map
                for (int j = 0; j < oneblock.racks.length; j++) {
                   ..
                }
      
      1. MAPREDUCE-2185.patch
        7 kB
        Ramkumar Vadali

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        191d 23h 51m 1 Ramkumar Vadali 23/May/11 20:04
        Patch Available Patch Available Resolved Resolved
        14d 2h 20m 1 Scott Chen 06/Jun/11 22:25
        Resolved Resolved Closed Closed
        161d 2h 23m 1 Arun C Murthy 15/Nov/11 00:48
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/)
        MAPREDUCE-2185. Fix infinite loop at creating splits using CombineFileInputFormat. (Ramkumar Vadali via schen)

        schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807
        Files :

        • /hadoop/mapreduce/trunk/CHANGES.txt
        • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
        • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/ ) MAPREDUCE-2185 . Fix infinite loop at creating splits using CombineFileInputFormat. (Ramkumar Vadali via schen) schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.23.0 [ 12315570 ]
        Resolution Fixed [ 1 ]
        Hide
        Scott Chen added a comment -

        I have committed this. Thanks Ram for the fix and dhruba for the review.

        Show
        Scott Chen added a comment - I have committed this. Thanks Ram for the fix and dhruba for the review.
        Hide
        dhruba borthakur added a comment -

        +1 code looks good to me.

        Show
        dhruba borthakur added a comment - +1 code looks good to me.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480153/MAPREDUCE-2185.patch
        against trunk revision 1126591.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480153/MAPREDUCE-2185.patch against trunk revision 1126591. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//console This message is automatically generated.
        Ramkumar Vadali made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Hairong Kuang [ hairong ] Ramkumar Vadali [ rvadali ]
        Ramkumar Vadali made changes -
        Attachment MAPREDUCE-2185.patch [ 12480153 ]
        Hide
        Ramkumar Vadali added a comment -

        For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as the rack location. This avoids the infinite loop later on in getMoreSplits()

        Show
        Ramkumar Vadali added a comment - For blocks that do not have hosts associated with them, use NetworkTopology.DEFAULT_RACK as the rack location. This avoids the infinite loop later on in getMoreSplits()
        Nigel Daley made changes -
        Field Original Value New Value
        Fix Version/s 0.22.0 [ 12314184 ]
        Hairong Kuang created issue -

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Hairong Kuang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development