[MAPREDUCE-452] tasktracker checkpointing capability

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Pete Wyckoff
• Created: 23/May/07 00:24
• Updated: 17/Jul/14 17:12
• Resolved: 17/Jul/14 17:12

Description

This relates to allowing a resource manager (e.g., hadoop on demand) to grow and (rarely) shrink jobs on the fly.

Growing is already supported. Shrinking could be done in 2 ways - (1) consider the machine dead and allow speculative execution to take care of it or (2) moving the existing map outputs from that machine somewhere else (another machine, dfs) - "task tracker checkpointing"

In the case of IO only intensive jobs, checkpointing the tasktracker doesn't do much for you. But, in the case of CPU or other scarce resource (e.g., a DB or Webpage cache...), the checkpointing could be very useful. The question is how often is this the case and how useful?

Issue Links

• is related to: MAPREDUCE-443 snapshot a map-reduce to DFS ... and restore