[HADOOP-3226] Run combiner when merging spills from map output - ASF JIRA

XML

Word

Printable

JSON

Hadoop Flags:

Incompatible change, Reviewed
Release Note:

Hide
Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls:
job.setCombineOnlyOnce(true);

Show
Changed policy for running combiner. The combiner may be run multiple times as the map's output is sorted and merged. Additionally, it may be run on the reduce side as data is merged. The old semantics are available in Hadoop 0.18 if the user calls: job.setCombineOnlyOnce(true);

When merging spills from the map, running the combiner should further diminish the volume of data we send to the reduce.