[MAPREDUCE-5952] LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.5.0
• Component/s: mr-am, mrv2
• Target Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Gera Shegalov
• Reporter: Gera Shegalov
• Created: 30/Jun/14 23:08
• Updated: 15/Aug/14 05:47
• Resolved: 16/Jul/14 21:34

Description

The javadoc comment for renameMapOutputForReduce incorrectly refers to a single map output directory, whereas this depends on LOCAL_DIRS.
mapOutIndex should be set to subMapOutputFile.getOutputIndexFile()

2014-06-30 14:48:35,574 WARN [uber-SubtaskRunner] org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local (uberized) 'child' : java.io.FileNotFoundException: File /Users/gshegalov/workspace/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.          TestMRJobs-localDir-nm-2_3/usercache/gshegalov/appcache/application_1404164272885_0001/output/file.out.index does not exist
  at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:517)
  at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:726)
  at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:507)
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)                      
  at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)                          
  at org.apache.hadoop.fs.RawLocalFileSystem.rename(RawLocalFileSystem.java:334)    
  at org.apache.hadoop.fs.ChecksumFileSystem.rename(ChecksumFileSystem.java:504)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.renameMapOutputForReduce(LocalContainerLauncher.java:471)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:370)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:292)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:178)
  at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:221)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)    
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)         
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)                       
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
  at java.lang.Thread.run(Thread.java:695)         

Issue Links

• is broken by: MAPREDUCE-5481 Enable uber jobs to have multiple reducers