Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5952

LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The javadoc comment for renameMapOutputForReduce incorrectly refers to a single map output directory, whereas this depends on LOCAL_DIRS.
      mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()

      2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.io.FileNotFoundException: File /Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.          TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)                      
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)                          
        at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)    
        at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)    
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)                       
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:695)         
      
      1. MAPREDUCE-5952.v04.patch
        9 kB
        Gera Shegalov
      2. MAPREDUCE-5952.v03.patch
        10 kB
        Gera Shegalov
      3. MAPREDUCE-5952.v02.patch
        7 kB
        Gera Shegalov
      4. MAPREDUCE-5952.v01.patch
        2 kB
        Gera Shegalov

        Issue Links

          Activity

          Hide
          Gera Shegalov added a comment -

          Found while working on the test for MAPREDUCE-207

          Show
          Gera Shegalov added a comment - Found while working on the test for MAPREDUCE-207
          Hide
          Gera Shegalov added a comment -

          v01 patch for review.

          Show
          Gera Shegalov added a comment - v01 patch for review.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The fix seems reasonable to me. Can we add a unit test?

          Show
          Karthik Kambatla (Inactive) added a comment - The fix seems reasonable to me. Can we add a unit test?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Gera Shegalov - can we iterate on this faster to unblock the release?

          Show
          Karthik Kambatla (Inactive) added a comment - Gera Shegalov - can we iterate on this faster to unblock the release?
          Hide
          Gera Shegalov added a comment -

          v02 with a targeted unit test.

          Show
          Gera Shegalov added a comment - v02 with a targeted unit test.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654958/MAPREDUCE-5952.v02.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654958/MAPREDUCE-5952.v02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.

          Show
          Karthik Kambatla (Inactive) added a comment - I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.
          Hide
          Gera Shegalov added a comment -

          I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.

          Karthik Kambatla, no that was my rebase mistake. I replaced private renameMapOutputForReduce by a 'protected static', but the old method definition remained in the file due to the merge error. Attaching v03 with the private renameMapOutputForReduce finally deleted.

          Show
          Gera Shegalov added a comment - I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code. Karthik Kambatla , no that was my rebase mistake. I replaced private renameMapOutputForReduce by a 'protected static', but the old method definition remained in the file due to the merge error. Attaching v03 with the private renameMapOutputForReduce finally deleted.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655152/MAPREDUCE-5952.v03.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655152/MAPREDUCE-5952.v03.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Looks like the new patch moves the method to a different line number in the file. Can we keep it in place for better history tracking? Sorry for the additional inconvenience.

          Show
          Karthik Kambatla (Inactive) added a comment - Looks like the new patch moves the method to a different line number in the file. Can we keep it in place for better history tracking? Sorry for the additional inconvenience.
          Hide
          Gera Shegalov added a comment -

          Karthik Kambatla, the reason the method was moved because it's static in nature (does not need any instance fields) and I needed to make it static for easier testability. EventHandler is not a static inner class. Making it static will also move other methods around. Here is v04 that avoids moving renameMapOutputForReduce but it will remain an instance method.

          Show
          Gera Shegalov added a comment - Karthik Kambatla , the reason the method was moved because it's static in nature (does not need any instance fields) and I needed to make it static for easier testability. EventHandler is not a static inner class. Making it static will also move other methods around. Here is v04 that avoids moving renameMapOutputForReduce but it will remain an instance method.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655673/MAPREDUCE-5952.v04.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655673/MAPREDUCE-5952.v04.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Gera Shegalov. Missed the fact that the method was moved out of the class. I agree with you making it static is a better approach; i.e., MAPREDUCE-5952.v03.patch.

          +1. Committing this shortly.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Gera Shegalov . Missed the fact that the method was moved out of the class. I agree with you making it static is a better approach; i.e., MAPREDUCE-5952 .v03.patch. +1. Committing this shortly.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Gera. Committed this to trunk, branch-2 and branch-2.5.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Gera. Committed this to trunk, branch-2 and branch-2.5.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5894/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5894/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/615/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/615/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1834/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1834/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1807/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1807/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development