Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3586

keep combiner backward compatible with earlier versions of hadoop

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.18.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In hadoop 16 and earlier, the combiner was guaranteed to run once and only once for each map. In 17 this compatibility was slightly broken: the combiner does not run if a single <K,V> occupies the entire sort buffer. In 18, this is further changed to where the combiner can be called multiple times on both map and reduce sides.

      This breaks Pig's current implementation of the combiner and it is not easy to fix in a short period of time.

      We would like to ask that for a way for an application to ask for a backward compatible behavior for some period of time until it can adjust to the new behavior.

        Attachments

        1. 3586-0.patch
          4 kB
          Christopher Douglas
        2. 3586-1.patch
          4 kB
          Christopher Douglas
        3. 3586-2.patch
          4 kB
          Christopher Douglas

          Issue Links

            Activity

              People

              • Assignee:
                cdouglas Christopher Douglas
                Reporter:
                olgan Olga Natkovich
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: