Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1287

Avoid calling Partitioner with only 1 reducer

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.22.0
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • For jobs with only one reducer, the Partitioner will no longer be called. Applications depending on Partitioners modifying records for single reducer jobs will need to move this functionality elsewhere.

    Description

      Partitioners are currently called for each record, even though all are destined for the same reduce.

      Attachments

        1. M1287-4.patch
          2 kB
          Christopher Douglas
        2. M1287-5.patch
          2 kB
          Christopher Douglas
        3. M1287-6.patch
          7 kB
          Christopher Douglas
        4. MAPREDUCE-1287.2.patch
          1 kB
          Ed Mazur
        5. MAPREDUCE-1287.3.patch
          1 kB
          Ed Mazur
        6. MAPREDUCE-1287.patch
          0.7 kB
          Ed Mazur

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-2632 Avoid calling the partitioner when the numReduceTasks is 1.

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              cdouglas Christopher Douglas
              edmazur Ed Mazur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: