Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6985

MapReduce native optimization does not work properly due to a shuffle error (LocalJobRunner)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reported by Shingo Furuyama:

      We confirmed that MapReduce native optimization (MAPREDUCE-2841) of Hadoop-3.0.0-beta1 does not work properly due to a shuffle error.

      2017-10-19 11:59:40,513 WARN mapred.Task: Could not find output size
      org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories
      	at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:489)
      	at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:166)
      	at org.apache.hadoop.mapred.MROutputFiles.getOutputFile(MROutputFiles.java:57)
      	at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1248)
      	at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1228)
      	at org.apache.hadoop.mapred.Task.done(Task.java:1174)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:345)
      	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      

      My build and run environment is below:
      CentOS: 7.3
      cmake: 3.6.3
      Maven: 3.5.0
      Java: 1.8.0_131
      Hadoop: 3.0.0-beta1
      mode: LocalJobRunner

        Attachments

        1. mr_native_optimization_error.log
          9 kB
          Takanobu Asanuma

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tasanuma Takanobu Asanuma
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: