[MAPREDUCE-4585] Checkpoint shuffle aggregation as map output - ASF JIRA

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: None
Fix Version/s: None
Component/s: task
Labels:
None

Target Version/s:

Description

Map output collected during the shuffle can be spilled and written as a composite of map outputs. Particularly if the job employs a combiner, this checkpoint can provide fault tolerance and improve job throughput by aggregating intermediate output. The latter is especially helpful for jobs with multiple waves of reduces.