Hadoop Common
  1. Hadoop Common
  2. HADOOP-3702

add support for chaining Maps in a single Map and after a Reduce [M*/RM*]

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.
      Show
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.

      Description

      On the same input, we usually need to run multiple Maps one after the other without no Reduce. We also have to run multiple Maps after the Reduce.

      If all pre-Reduce Maps are chained together and run as a single Map a significant amount of Disk I/O will be avoided.

      Similarly all post-Reduce Maps can be chained together and run in the Reduce phase after the Reduce.

      1. Hadoop-3702.patch
        60 kB
        Enis Soztutar
      2. patch3702.txt
        60 kB
        Alejandro Abdelnur
      3. patch3702.txt
        59 kB
        Alejandro Abdelnur
      4. patch3702.txt
        59 kB
        Alejandro Abdelnur
      5. patch3702.txt
        58 kB
        Alejandro Abdelnur
      6. patch3702.txt
        57 kB
        Alejandro Abdelnur
      7. patch3702.txt
        56 kB
        Alejandro Abdelnur
      8. patch3702.txt
        54 kB
        Alejandro Abdelnur
      9. patch3702.txt
        53 kB
        Alejandro Abdelnur
      10. patch3702.txt
        47 kB
        Alejandro Abdelnur
      11. patch3702.txt
        45 kB
        Alejandro Abdelnur
      12. patch3702.txt
        33 kB
        Alejandro Abdelnur
      13. patch3702.txt
        30 kB
        Alejandro Abdelnur
      14. patch3702.txt
        30 kB
        Alejandro Abdelnur

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Alejandro Abdelnur
              Reporter:
              Alejandro Abdelnur
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development