Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5952

LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The javadoc comment for renameMapOutputForReduce incorrectly refers to a single map output directory, whereas this depends on LOCAL_DIRS.
      mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()

      2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.io.FileNotFoundException: File /Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.          TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)                      
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)                          
        at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)    
        at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
        at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)    
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)                       
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:695)         
      
      1. MAPREDUCE-5952.v01.patch
        2 kB
        Gera Shegalov
      2. MAPREDUCE-5952.v02.patch
        7 kB
        Gera Shegalov
      3. MAPREDUCE-5952.v03.patch
        10 kB
        Gera Shegalov
      4. MAPREDUCE-5952.v04.patch
        9 kB
        Gera Shegalov

        Issue Links

          Activity

          Gera Shegalov created issue -
          Gera Shegalov made changes -
          Field Original Value New Value
          Link This issue is broken by MAPREDUCE-5481 [ MAPREDUCE-5481 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5952.v01.patch [ 12653262 ]
          Gera Shegalov made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.5.0 [ 12326265 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5952.v02.patch [ 12654958 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5952.v03.patch [ 12655152 ]
          Gera Shegalov made changes -
          Attachment MAPREDUCE-5952.v04.patch [ 12655673 ]
          Karthik Kambatla (Inactive) made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.5.0 [ 12326265 ]
          Resolution Fixed [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Gera Shegalov
              Reporter:
              Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development