Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3310

Custom grouping comparator cannot be set for Combiners

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 1.3.0, 2.3.0
    • Component/s: client
    • Labels:
      None
    • Environment:

      All

    • Hadoop Flags:
      Reviewed

      Description

      Combiners are often described as 'Reducers running on the Map side'.

      As Reducers, Combiners are fed <K,

      {V}>, where {V}

      is built by grouping values associated with the 'same' key.

      For Reducers, the comparator used for grouping values can be set independently of that used to sort the keys (using Job.setGroupingComparatorClass).

      Such a configuration is not possible for Combiners, meaning some things done in Reducers cannot be done in Combiners (such as secondary sort).

      It would be handy to have a Job.setCombinerGroupingComparatorClass method that would allow the setting of the grouping comparator used when applying a Combiner.

      1. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      2. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      3. MAPREDUCE-3310-branch-1.patch
        18 kB
        Alejandro Abdelnur
      4. MAPREDUCE-3310-trunk.patch
        28 kB
        Alejandro Abdelnur
      5. MAPREDUCE-3310-branch-1.patch
        21 kB
        Alejandro Abdelnur
      6. MAPREDUCE-3310-trunk.patch
        29 kB
        Alejandro Abdelnur

        Activity

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Mathias Herberts
          • Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development