Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
From what I can tell, the Combiner just aggregates data from a single map task. It would be useful, especially during map-only jobs, to have a combiner that aggregates data from all the map tasks on a given machine. My use case for this is to vertically partition a set of records which start out in the same files. By doing this in a map-only task, way too many files are created (About 50 files are created per input split). By pumping all the data through a reducer, a lot of unnecessary overhead occurs. With the proposed feature, I would get 50*number of machines files rather than 50*number of input splits files for this use case.