Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6357

MultipleOutputs.write() API should document that output committing is not utilized when input path is absolute

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After spending the afternoon debugging a user job where reduce tasks were failing on retry with the below exception, I think it would be worthwhile to add a note in the MultipleOutputs.write() documentation, saying that absolute paths may cause improper execution of tasks on retry or when MR speculative execution is enabled.

      2015-04-28 23:13:10,452 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: File already exists:wasb://full20150320@bgtstoragefull.blob.core.windows.net/user/hadoop/some/path/block-r-00299.bz2
             at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1354)
             at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
             at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
             at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
             at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
             at org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:135)
             at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:475)
             at org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:433)
             at com.ancestry.bigtree.hadoop.LevelReducer.processValue(LevelReducer.java:91)
             at com.ancestry.bigtree.hadoop.LevelReducer.reduce(LevelReducer.java:69)
             at com.ancestry.bigtree.hadoop.LevelReducer.reduce(LevelReducer.java:14)
             at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
             at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
             at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
             at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
             at java.security.AccessController.doPrivileged(Native Method)
             at javax.security.auth.Subject.doAs(Subject.java:415)
             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
             at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
      

      As discussed in MAPREDUCE-3772, when the baseOutputPath passed to MultipleOutputs.write() is an absolute path (or more precisely a path that resolves outside of the job output-dir), the concept of output committing is not utilized.

      In this case, the user read thru the MultipleOutputs docs and was assuming that everything will be working fine, as there are blog posts saying that MultipleOutputs does handle output commit.

        Attachments

          Activity

            People

            • Assignee:
              cotedm Dustin Cote
              Reporter:
              ivanmi Ivan Mitic
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: