Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5221 Reduce side Combiner is not used when using the new API
  3. MAPREDUCE-5294

Shuffle#MergeManager should support org.apache.hadoop.mapreduce.Reducer

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.0-beta, 2.0.5-alpha
    • None
    • mrv2
    • None
    • Reviewed

    Description

      Shuffle#MergeManager only accepts org.apache.hadoop.mapred.Reducer currently. Because of this, Reduce-side Combiner is not used when using the new API, and just ignored. By supporting it and using the feature from ReduceTask, Reduce-side combiner can be enabled with new API. Please see MAPREDUCE-5221 for more detail.

      Attachments

        1. MAPREDUCE-5294.1.patch
          10 kB
          Tsuyoshi Ozawa
        2. MAPREDUCE-5294.2.patch
          11 kB
          Tsuyoshi Ozawa
        3. MAPREDUCE-5294.3.patch
          12 kB
          Tsuyoshi Ozawa

        Issue Links

        blocks

        Sub-task - The sub-task of the issue MAPREDUCE-5295 ShuffleConsumerPlugin.Context should remove unused combiner-related methods and fields

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5221 Reduce side Combiner is not used when using the new API

        • Major - Major loss of function.
        • In Progress

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ozawa Tsuyoshi Ozawa
            ozawa Tsuyoshi Ozawa
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment