[MAPREDUCE-5176] Preemptable annotations (to support preemption in MR)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0-beta
• Component/s: mrv2
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Carlo Curino
• Reporter: Carlo Curino
• Created: 24/Apr/13 00:02
• Updated: 15/Jul/15 23:32
• Resolved: 31/May/13 06:08

Description

Proposing a patch that introduces a new annotation @Checkpointable that represents to the framework property of user-supplied classes (e.g., Reducer, OutputCommiter). The intended semantics is that a tagged class is safe to be preempted between invocations.

(this is in spirit similar to the Output Contracts of Nephele/PACT )

Issue Links

• is part of: MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks
• is related to: YARN-567 RM changes to support preemption for FairScheduler and CapacityScheduler
• is related to: YARN-568 FairScheduler: support for work-preserving preemption
• is related to: YARN-569 CapacityScheduler: support for preemption (using a capacity monitor)
• relates to: MAPREDUCE-5189 Basic AM changes to support preemption requests (per YARN-45)
• relates to: MAPREDUCE-5196 CheckpointAMPreemptionPolicy implements preemption in MR AM via checkpointing
• relates to: MAPREDUCE-6434 Add support for PartialFileOutputCommiter when checkpointing is an option during preemption