Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5221 Reduce side Combiner is not used when using the new API
  3. MAPREDUCE-5295

ShuffleConsumerPlugin.Context should remove unused combiner-related methods and fields

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.0-beta, 2.0.5-alpha
    • None
    • None
    • None

    Description

      ShuffleConsumerPlugin.Context only supports org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce side Combiner is not used when using the new API, and just ignored. Please see MAPREDUCE-5221 for more detail.

      (Update Jun. 7th, 2013) The latest version of MergeManagerImpl in MAPREDUCE-5294 never calls ShuffleConsumerPlugin.Context#getCombinerClass() as masokan pointed out. Because of this, combinerClass field and getCombinerClass() method should be removed from ShuffleConsumerPlugin.Context.

      Attachments

        1. MAPREDUCE-5295.3.patch
          6 kB
          Tsuyoshi Ozawa
        2. MAPREDUCE-5295.2.patch
          7 kB
          Tsuyoshi Ozawa
        3. MAPREDUCE-5295.1.patch
          7 kB
          Tsuyoshi Ozawa

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5221 Reduce side Combiner is not used when using the new API

          • Major - Major loss of function.
          • In Progress
          is blocked by

          Sub-task - The sub-task of the issue MAPREDUCE-5294 Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              ozawa Tsuyoshi Ozawa
              ozawa Tsuyoshi Ozawa
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: