Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4808

Refactor MapOutput and MergeManager to facilitate reuse by Shuffle implementations

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.3-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Now that Shuffle is pluggable (MAPREDUCE-4049), it would be convenient for alternate implementations to be able to reuse portions of the default implementation.

      This would come with the strong caveat that these classes are LimitedPrivate and Unstable.

      1. mapreduce-4808.patch
        39 kB
        Mariappan Asokan
      2. COMBO-mapreduce-4809-4812-4808.patch
        84 kB
        Mariappan Asokan
      3. mapreduce-4808.patch
        67 kB
        Mariappan Asokan
      4. mapreduce-4808.patch
        67 kB
        Mariappan Asokan
      5. MergeManagerPlugin.pdf
        47 kB
        Mariappan Asokan
      6. mapreduce-4808.patch
        71 kB
        Mariappan Asokan
      7. mapreduce-4808.patch
        78 kB
        Mariappan Asokan
      8. mapreduce-4808.patch
        61 kB
        Mariappan Asokan
      9. mapreduce-4808.patch
        62 kB
        Mariappan Asokan
      10. MR-4808.patch
        51 kB
        Alejandro Abdelnur
      11. mapreduce-4808.patch
        45 kB
        Mariappan Asokan
      12. M4808-0.patch
        103 kB
        Chris Douglas
      13. M4808-1.patch
        103 kB
        Chris Douglas

        Issue Links

          Activity

          Arun C Murthy created issue -
          Mariappan Asokan made changes -
          Field Original Value New Value
          Attachment mapreduce-4808.patch [ 12554786 ]
          Attachment COMBO-mapreduce-4809-4812-4808.patch [ 12554787 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.0.2-alpha [ 12322471 ]
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Mariappan Asokan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Mariappan Asokan made changes -
          Attachment COMBO-mapreduce-4809-4812-4808.patch [ 12554787 ]
          Mariappan Asokan made changes -
          Attachment COMBO-mapreduce-4809-4812-4808.patch [ 12554898 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Alejandro Abdelnur made changes -
          Parent MAPREDUCE-2454 [ 12505228 ]
          Issue Type Sub-task [ 7 ] New Feature [ 2 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12561370 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12561532 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Mariappan Asokan made changes -
          Attachment MAPREDUCE-4808.patch [ 12562029 ]
          Attachment MergeManagerPlugin.pdf [ 12562030 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Mariappan Asokan made changes -
          Attachment MAPREDUCE-4808.patch [ 12562029 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12562034 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12562292 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12564997 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12565016 ]
          Alejandro Abdelnur made changes -
          Summary Allow reduce-side merge to be pluggable Refactor MapOutput and MergeManager to facilitate reuse by Shuffle implementations
          Description Allow reduce-side merge to be pluggable for MAPREDUCE-2454 Now that Shuffle is pluggable (MAPREDUCE-4049), it would be convenient for alternate implementations to be able to reuse portions of the default implementation.

          This would come with the strong caveat that these classes are LimitedPrivate and Unstable.
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Arun C Murthy made changes -
          Affects Version/s 2.0.2-alpha [ 12322471 ]
          Alejandro Abdelnur made changes -
          Attachment MR-4808.patch [ 12565391 ]
          Mariappan Asokan made changes -
          Attachment mapreduce-4808.patch [ 12565547 ]
          Chris Douglas made changes -
          Attachment M4808-0.patch [ 12565578 ]
          Mariappan Asokan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Attachment M4808-1.patch [ 12565595 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Alejandro Abdelnur made changes -
          Fix Version/s 2.0.3-alpha [ 12323275 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to MAPREDUCE-6332 [ MAPREDUCE-6332 ]

            People

            • Assignee:
              Mariappan Asokan
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development