Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3310

Custom grouping comparator cannot be set for Combiners

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 1.3.0, 2.3.0
    • Component/s: client
    • Labels:
      None
    • Environment:

      All

    • Hadoop Flags:
      Reviewed

      Description

      Combiners are often described as 'Reducers running on the Map side'.

      As Reducers, Combiners are fed <K,

      {V}>, where {V}

      is built by grouping values associated with the 'same' key.

      For Reducers, the comparator used for grouping values can be set independently of that used to sort the keys (using Job.setGroupingComparatorClass).

      Such a configuration is not possible for Combiners, meaning some things done in Reducers cannot be done in Combiners (such as secondary sort).

      It would be handy to have a Job.setCombinerGroupingComparatorClass method that would allow the setting of the grouping comparator used when applying a Combiner.

      1. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      2. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      3. MAPREDUCE-3310-branch-1.patch
        18 kB
        Alejandro Abdelnur
      4. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      5. MAPREDUCE-3310-branch-1.patch
        21 kB
        Alejandro Abdelnur
      6. MAPREDUCE-3310-trunk.patch
        29 kB
        Alejandro Abdelnur

        Activity

        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.3.0 [ 12325257 ]
        Fix Version/s 2.4.0 [ 12324590 ]
        Alejandro Abdelnur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 1.3.0 [ 12324153 ]
        Fix Version/s 2.4.0 [ 12324590 ]
        Resolution Fixed [ 1 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-trunk.patch [ 12620265 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-branch-1.patch [ 12620264 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-trunk.patch [ 12614010 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-branch-1.patch [ 12614009 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-trunk.patch [ 12614008 ]
        Alejandro Abdelnur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Alejandro Abdelnur made changes -
        Attachment MAPREDUCE-3310-trunk.patch [ 12613990 ]
        Alejandro Abdelnur made changes -
        Assignee Alejandro Abdelnur [ tucu00 ]
        Harsh J made changes -
        Field Original Value New Value
        Affects Version/s 0.21.0 [ 12314045 ]
        Affects Version/s 0.20.2 [ 12314205 ]
        Affects Version/s 0.20.203.0 [ 12316151 ]
        Affects Version/s 0.20.205.0 [ 12316391 ]
        Mathias Herberts created issue -

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Mathias Herberts
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development