Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3149

supporting multiple outputs for M/R jobs

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced MultipleOutputs class so Map/Reduce jobs can write data to different output files. Each output can use a different OutputFormat. Outpufiles are created within the job output directory. FileOutputFormat.getPathForCustomFile() creates a filename under the outputdir that is named with the task ID and task type (i.e. myfile-r-00001).
      Show
      Introduced MultipleOutputs class so Map/Reduce jobs can write data to different output files. Each output can use a different OutputFormat. Outpufiles are created within the job output directory. FileOutputFormat.getPathForCustomFile() creates a filename under the outputdir that is named with the task ID and task type (i.e. myfile-r-00001).

      Description

      The outputcollector supports writing data to a single output, the 'part' files in the output path.

      We found quite common that our M/R jobs have to write data to different output. For example when classifying data as NEW, UPDATE, DELETE, NO-CHANGE to later do different processing on it.

      Handling the initialization of additional outputs from within the M/R code complicates the code and is counter intuitive with the notion of job configuration.

      It would be desirable to:

      1. Configure the additional outputs in the jobconf, potentially specifying different outputformats, key and value classes for each one.
      2. Write to the additional outputs in a similar way as data is written to the outputcollector.
      3. Support the speculative execution semantics for the output files, only visible in the final output for promoted tasks.

      To support multiple outputs the following classes would be added to mapred/lib:

      • MOJobConf : extends JobConf adding methods to define named outputs (name, outputformat, key class, value class)
      • MOOutputCollector : extends OutputCollector adding a collect(String outputName, WritableComparable key, Writable value) method.
      • MOMapper and MOReducer: implement Mapper and Reducer adding a new configure, map and reduce signature that take the corresponding MO classes and performs the proper initialization.

      The data flow behavior would be: key/values written to the default (unnamed) output (using the original OutputCollector collect signature) take part of the shuffle/sort/reduce processing phases. key/values written to a named output from within a map don't.

      The named output files would be named using the task type and task ID to avoid collision among tasks (i.e. 'new-m-00002' and 'new-r-00001').

      Together with the setInputPathFilter feature introduced by HADOOP-2055 it would become very easy to chain jobs working on particular named outputs within a single directory.

      We are using heavily this pattern and it greatly simplified our M/R code as well as chaining different M/R.

      We wanted to contribute this back to Hadoop as we think is a generic feature many could benefit from.

        Attachments

        1. patch3149.txt
          32 kB
          Alejandro Abdelnur
        2. patch3149.txt
          32 kB
          Alejandro Abdelnur
        3. patch3149.txt
          32 kB
          Alejandro Abdelnur
        4. patch3149.txt
          32 kB
          Alejandro Abdelnur
        5. patch3149.txt
          33 kB
          Alejandro Abdelnur
        6. patch3149.txt
          33 kB
          Alejandro Abdelnur
        7. patch3149.txt
          19 kB
          Alejandro Abdelnur
        8. patch3149.txt
          19 kB
          Alejandro Abdelnur
        9. patch3149.txt
          19 kB
          Alejandro Abdelnur
        10. patch3149.txt
          20 kB
          Alejandro Abdelnur
        11. patch3149.txt
          19 kB
          Alejandro Abdelnur
        12. patch3149.txt
          19 kB
          Alejandro Abdelnur
        13. patch3149.txt
          22 kB
          Alejandro Abdelnur
        14. patch3149.txt
          33 kB
          Alejandro Abdelnur
        15. patch3149.txt
          33 kB
          Alejandro Abdelnur
        16. patch3149.txt
          29 kB
          Alejandro Abdelnur

          Issue Links

            Activity

              People

              • Assignee:
                tucu00 Alejandro Abdelnur
                Reporter:
                tucu00 Alejandro Abdelnur
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: