[MAPREDUCE-6444] Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Augusto Souza
• Reporter: Augusto Souza
• Created: 05/Aug/15 22:52
• Updated: 28/Aug/15 05:54

Description

Patch to add classes to provide a checkpointable version of shuffle and reduce context. Also, add a classe which makes possible to save a checkpoint of a task in HDFS. Those classes are respectively: CheckpointableShuffle, CheckpointableReduceContextImpl and TaskCheckpointManager.

This improvement also helps in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts.

Issue Links

• is depended upon by: MAPREDUCE-5269 Preemption of Reducer (and Shuffle) via checkpointing
• is related to: MAPREDUCE-6434 Add support for PartialFileOutputCommiter when checkpointing is an option during preemption