Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
3.0.0-beta1
-
None
-
None
-
None
Description
Reported by Shingo Furuyama:
We confirmed that MapReduce native optimization (MAPREDUCE-2841) of Hadoop-3.0.0-beta1 does not work properly due to a shuffle error.
2017-10-19 11:59:40,513 WARN mapred.Task: Could not find output size org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:489) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:166) at org.apache.hadoop.mapred.MROutputFiles.getOutputFile(MROutputFiles.java:57) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:1248) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:1228) at org.apache.hadoop.mapred.Task.done(Task.java:1174) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:345) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)
My build and run environment is below:
CentOS: 7.3
cmake: 3.6.3
Maven: 3.5.0
Java: 1.8.0_131
Hadoop: 3.0.0-beta1
mode: LocalJobRunner
Attachments
Attachments
Issue Links
- relates to
-
MAPREDUCE-2841 Task level native optimization
- Resolved