Hadoop Common
  1. Hadoop Common
  2. HADOOP-2399

Input key and value to combiner and reducer should be reused

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.15.1
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The key and value objects that are given to the Combiner and Reducer are now reused between calls. This is much more efficient, but the user can not assume the objects are constant.

      Description

      Currently, the input key and value are recreated on every iteration for input to the combiner and reducer. It would speed up the system substantially if we reused the keys and values. The down side of doing it, is that it may break applications that count on holding references to previous keys and values, but I think it is worth doing.

      1. 2399-4.patch
        16 kB
        Owen O'Malley
      2. 2399-3.patch
        15 kB
        Owen O'Malley

        Issue Links

          Activity

          Owen O'Malley created issue -
          Owen O'Malley made changes -
          Field Original Value New Value
          Attachment reuse-obj.patch [ 12372937 ]
          Nigel Daley made changes -
          Fix Version/s 0.16.0 [ 12312740 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Owen O'Malley made changes -
          Attachment reuse-obj-2.patch [ 12375629 ]
          Owen O'Malley made changes -
          Attachment 2399-3.patch [ 12375966 ]
          Owen O'Malley made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Fix Version/s 0.17.0 [ 12312913 ]
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Owen O'Malley made changes -
          Attachment reuse-obj-2.patch [ 12375629 ]
          Owen O'Malley made changes -
          Attachment reuse-obj.patch [ 12372937 ]
          Owen O'Malley made changes -
          Attachment 2399-4.patch [ 12377559 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Devaraj Das made changes -
          Resolution Fixed [ 1 ]
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Tom White made changes -
          Link This issue relates to HADOOP-2997 [ HADOOP-2997 ]
          Owen O'Malley made changes -
          Release Note The key and value objects that are given to the Combiner and Reducer are now reused between calls. This is much more efficient, but the user can not assume the objects are constant.
          Hadoop Flags [Incompatible change, Reviewed]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]

            People

            • Assignee:
              Owen O'Malley
              Reporter:
              Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development