Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2399

Input key and value to combiner and reducer should be reused

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.15.1
    • 0.17.0
    • None
    • None
    • Incompatible change, Reviewed
    • The key and value objects that are given to the Combiner and Reducer are now reused between calls. This is much more efficient, but the user can not assume the objects are constant.

    Description

      Currently, the input key and value are recreated on every iteration for input to the combiner and reducer. It would speed up the system substantially if we reused the keys and values. The down side of doing it, is that it may break applications that count on holding references to previous keys and values, but I think it is worth doing.

      Attachments

        1. 2399-4.patch
          16 kB
          Owen O'Malley
        2. 2399-3.patch
          15 kB
          Owen O'Malley

        Issue Links

          relates to

          Test - A new unit, integration or system test. HADOOP-2997 Add test for non-writable serializer

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              omalley Owen O'Malley
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: