Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3702

add support for chaining Maps in a single Map and after a Reduce [M*/RM*]

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.19.0
    • None
    • None
    • all

    • Reviewed
    • Hide
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.
      Show
      Introduced ChainMapper and the ChainReducer classes to allow composing chains of Maps and Reduces in a single Map/Reduce job, something like MAP+ REDUCE MAP*.

    Description

      On the same input, we usually need to run multiple Maps one after the other without no Reduce. We also have to run multiple Maps after the Reduce.

      If all pre-Reduce Maps are chained together and run as a single Map a significant amount of Disk I/O will be avoided.

      Similarly all post-Reduce Maps can be chained together and run in the Reduce phase after the Reduce.

      Attachments

        1. Hadoop-3702.patch
          60 kB
          Enis Soztutar
        2. patch3702.txt
          60 kB
          Alejandro Abdelnur
        3. patch3702.txt
          59 kB
          Alejandro Abdelnur
        4. patch3702.txt
          59 kB
          Alejandro Abdelnur
        5. patch3702.txt
          58 kB
          Alejandro Abdelnur
        6. patch3702.txt
          57 kB
          Alejandro Abdelnur
        7. patch3702.txt
          56 kB
          Alejandro Abdelnur
        8. patch3702.txt
          54 kB
          Alejandro Abdelnur
        9. patch3702.txt
          53 kB
          Alejandro Abdelnur
        10. patch3702.txt
          47 kB
          Alejandro Abdelnur
        11. patch3702.txt
          45 kB
          Alejandro Abdelnur
        12. patch3702.txt
          33 kB
          Alejandro Abdelnur
        13. patch3702.txt
          30 kB
          Alejandro Abdelnur
        14. patch3702.txt
          30 kB
          Alejandro Abdelnur

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tucu00 Alejandro Abdelnur
            tucu00 Alejandro Abdelnur
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment