Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5952

LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The javadoc comment for renameMapOutputForReduce incorrectly refers to a single map output directory, whereas this depends on LOCAL_DIRS.
      mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()

      2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.io.FileNotFoundException: File /Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.          TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)                      
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)                          
        at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)    
        at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)    
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)                       
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:695)         
      
      1. MAPREDUCE-5952.v04.patch
        9 kB
        Gera Shegalov
      2. MAPREDUCE-5952.v03.patch
        10 kB
        Gera Shegalov
      3. MAPREDUCE-5952.v02.patch
        7 kB
        Gera Shegalov
      4. MAPREDUCE-5952.v01.patch
        2 kB
        Gera Shegalov

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1807/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1807 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1807/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1834/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1834 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1834/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/615/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #615 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/615/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5894/)
          MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196)

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5894 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5894/ ) MAPREDUCE-5952 . LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1611196 ) /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestLocalContainerLauncher.java
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Gera. Committed this to trunk, branch-2 and branch-2.5.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Gera. Committed this to trunk, branch-2 and branch-2.5.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Gera Shegalov. Missed the fact that the method was moved out of the class. I agree with you making it static is a better approach; i.e., MAPREDUCE-5952.v03.patch.

          +1. Committing this shortly.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Gera Shegalov . Missed the fact that the method was moved out of the class. I agree with you making it static is a better approach; i.e., MAPREDUCE-5952 .v03.patch. +1. Committing this shortly.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655673/MAPREDUCE-5952.v04.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:

          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf
          org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655673/MAPREDUCE-5952.v04.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app: org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesAttempts org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobConf org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4733//console This message is automatically generated.
          Hide
          Gera Shegalov added a comment -

          Karthik Kambatla, the reason the method was moved because it's static in nature (does not need any instance fields) and I needed to make it static for easier testability. EventHandler is not a static inner class. Making it static will also move other methods around. Here is v04 that avoids moving renameMapOutputForReduce but it will remain an instance method.

          Show
          Gera Shegalov added a comment - Karthik Kambatla , the reason the method was moved because it's static in nature (does not need any instance fields) and I needed to make it static for easier testability. EventHandler is not a static inner class. Making it static will also move other methods around. Here is v04 that avoids moving renameMapOutputForReduce but it will remain an instance method.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Looks like the new patch moves the method to a different line number in the file. Can we keep it in place for better history tracking? Sorry for the additional inconvenience.

          Show
          Karthik Kambatla (Inactive) added a comment - Looks like the new patch moves the method to a different line number in the file. Can we keep it in place for better history tracking? Sorry for the additional inconvenience.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12655152/MAPREDUCE-5952.v03.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12655152/MAPREDUCE-5952.v03.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4727//console This message is automatically generated.
          Hide
          Gera Shegalov added a comment -

          I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.

          Karthik Kambatla, no that was my rebase mistake. I replaced private renameMapOutputForReduce by a 'protected static', but the old method definition remained in the file due to the merge error. Attaching v03 with the private renameMapOutputForReduce finally deleted.

          Show
          Gera Shegalov added a comment - I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code. Karthik Kambatla , no that was my rebase mistake. I replaced private renameMapOutputForReduce by a 'protected static', but the old method definition remained in the file due to the merge error. Attaching v03 with the private renameMapOutputForReduce finally deleted.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.

          Show
          Karthik Kambatla (Inactive) added a comment - I must be missing something here. LocalContainerLauncher#renameMapOutputForReduce isn't called anywhere in non-test code.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12654958/MAPREDUCE-5952.v02.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12654958/MAPREDUCE-5952.v02.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4723//console This message is automatically generated.
          Hide
          Gera Shegalov added a comment -

          v02 with a targeted unit test.

          Show
          Gera Shegalov added a comment - v02 with a targeted unit test.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Gera Shegalov - can we iterate on this faster to unblock the release?

          Show
          Karthik Kambatla (Inactive) added a comment - Gera Shegalov - can we iterate on this faster to unblock the release?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The fix seems reasonable to me. Can we add a unit test?

          Show
          Karthik Kambatla (Inactive) added a comment - The fix seems reasonable to me. Can we add a unit test?
          Hide
          Gera Shegalov added a comment -

          v01 patch for review.

          Show
          Gera Shegalov added a comment - v01 patch for review.
          Hide
          Gera Shegalov added a comment -

          Found while working on the test for MAPREDUCE-207

          Show
          Gera Shegalov added a comment - Found while working on the test for MAPREDUCE-207

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development