Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1287

Avoid calling Partitioner with only 1 reducer

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      For jobs with only one reducer, the Partitioner will no longer be called. Applications depending on Partitioners modifying records for single reducer jobs will need to move this functionality elsewhere.

      Description

      Partitioners are currently called for each record, even though all are destined for the same reduce.

      1. M1287-6.patch
        7 kB
        Chris Douglas
      2. M1287-5.patch
        2 kB
        Chris Douglas
      3. M1287-4.patch
        2 kB
        Chris Douglas
      4. MAPREDUCE-1287.3.patch
        1 kB
        Ed Mazur
      5. MAPREDUCE-1287.2.patch
        1 kB
        Ed Mazur
      6. MAPREDUCE-1287.patch
        0.7 kB
        Ed Mazur

        Issue Links

          Activity

          Ed Mazur created issue -
          Ed Mazur made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1287.patch [ 12427522 ]
          Ed Mazur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ed Mazur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ed Mazur made changes -
          Attachment MAPREDUCE-1287.2.patch [ 12427523 ]
          Ed Mazur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ed Mazur made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Ed Mazur made changes -
          Attachment MAPREDUCE-1287.3.patch [ 12427524 ]
          Ed Mazur made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Tom White made changes -
          Assignee Ed Mazur [ edmazur ]
          Chris Douglas made changes -
          Attachment M1287-4.patch [ 12428502 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Attachment M1287-5.patch [ 12429728 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Summary HashPartitioner calls hashCode() when there is only 1 reducer Avoid calling Partitioner with only 1 reducer
          Release Note For jobs with only one reducer, the Partitioner will no longer be called. Applications depending on Partitioners modifying records for single reducer jobs will need to move this functionality elsewhere.
          Description HashPartitioner could be optimized to not call the key's hashCode() if there is only 1 reducer. Partitioners are currently called for each record, even though all are destined for the same reduce.
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Chris Douglas made changes -
          Attachment M1287-6.patch [ 12429732 ]
          Chris Douglas made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Resolution Fixed [ 1 ]
          Chris Douglas made changes -
          Assignee Ed Mazur [ edmazur ] Chris Douglas [ chris.douglas ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Harsh J made changes -
          Link This issue is duplicated by MAPREDUCE-2632 [ MAPREDUCE-2632 ]
          Harsh J made changes -
          Link This issue is duplicated by MAPREDUCE-2632 [ MAPREDUCE-2632 ]
          Harsh J made changes -
          Link This issue is part of MAPREDUCE-2632 [ MAPREDUCE-2632 ]

            People

            • Assignee:
              Chris Douglas
              Reporter:
              Ed Mazur
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development