Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3702

add support for chaining Maps in a single Map and after a Reduce [M*/RM*]

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      all

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.
      Show
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.

      Description

      On the same input, we usually need to run multiple Maps one after the other without no Reduce. We also have to run multiple Maps after the Reduce.

      If all pre-Reduce Maps are chained together and run as a single Map a significant amount of Disk I/O will be avoided.

      Similarly all post-Reduce Maps can be chained together and run in the Reduce phase after the Reduce.

        Attachments

        1. Hadoop-3702.patch
          60 kB
          Enis Soztutar
        2. patch3702.txt
          60 kB
          Alejandro Abdelnur
        3. patch3702.txt
          59 kB
          Alejandro Abdelnur
        4. patch3702.txt
          59 kB
          Alejandro Abdelnur
        5. patch3702.txt
          58 kB
          Alejandro Abdelnur
        6. patch3702.txt
          57 kB
          Alejandro Abdelnur
        7. patch3702.txt
          56 kB
          Alejandro Abdelnur
        8. patch3702.txt
          54 kB
          Alejandro Abdelnur
        9. patch3702.txt
          53 kB
          Alejandro Abdelnur
        10. patch3702.txt
          47 kB
          Alejandro Abdelnur
        11. patch3702.txt
          45 kB
          Alejandro Abdelnur
        12. patch3702.txt
          33 kB
          Alejandro Abdelnur
        13. patch3702.txt
          30 kB
          Alejandro Abdelnur
        14. patch3702.txt
          30 kB
          Alejandro Abdelnur

          Issue Links

            Activity

              People

              • Assignee:
                tucu00 Alejandro Abdelnur
                Reporter:
                tucu00 Alejandro Abdelnur
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: