Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5051

Combiner not used when NUM_REDUCES=0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
      None
    • Environment:

      CDH4.1.2 MR1

      Description

      We have a M/R job that use Mapper + Combiner but have nothing to do in Reducer :
      Bulk indexing of HBase data in ElasticSearch,
      Map output is K / V : #bulk / json_data_to_be_indexed.

      So job is launched maps work, combiners index and a reducer is created for nothing (sometimes waiting for other M/R job to free a tasktracker slot for reducer cf. MAPREDUCE-5019 )

      When we put ```job.setNumReduceTasks(0);``` in our job .run(), mapper are started but combiner are not used.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dam_ned Damien Hardy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: