[MAPREDUCE-6357] MultipleOutputs.write() API should document that output committing is not utilized when input path is absolute

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: documentation
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Dustin Cote
• Reporter: Ivan Mitic
• Created: 05/May/15 18:11
• Updated: 30/Aug/16 01:17
• Resolved: 21/Aug/15 01:45

Description

After spending the afternoon debugging a user job where reduce tasks were failing on retry with the below exception, I think it would be worthwhile to add a note in the MultipleOutputs.write() documentation, saying that absolute paths may cause improper execution of tasks on retry or when MR speculative execution is enabled.

2015-04-28 23:13:10,452 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: File already exists:wasb://[e-mail]/user/hadoop/some/path/block-r-00299.bz2
       at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1354)
       at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
       at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
       at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
       at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:475)
       at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:433)
       at com.ancestry.bigtree.hadoop.LevelReducer.processValue(LevelReducer.java:91)
       at com.ancestry.bigtree.hadoop.LevelReducer.reduce(LevelReducer.java:69)
       at com.ancestry.bigtree.hadoop.LevelReducer.reduce(LevelReducer.java:14)
       at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
       at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
       at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
       at java.security.AccessController.doPrivileged(Native Method)
       at javax.security.auth.Subject.doAs(Subject.java:415)
       at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

As discussed in MAPREDUCE-3772, when the baseOutputPath passed to MultipleOutputs.write() is an absolute path (or more precisely a path that resolves outside of the job output-dir), the concept of output committing is not utilized.

In this case, the user read thru the MultipleOutputs docs and was assuming that everything will be working fine, as there are blog posts saying that MultipleOutputs does handle output commit.