[MAPREDUCE-6608] Work Preserving AM Restart for MapReduce

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Target Version/s: MAPREDUCE-6608
• Assignee: Srikanth Sampath
• Reporter: Srikanth Sampath
• Created: 24/Jul/15 13:40
• Updated: 25/Oct/19 20:27

Description

Providing a framework for work preserving AM is achieved in YARN-1489. We would like to take advantage of this for MapReduce(MR) applications. There are some challenges which have been described in the attached document and few options discussed. We solicit feedback from the community.

Issue Links

• depends upon: MAPREDUCE-5485 Allow repeating job commit by extending OutputCommitter API
• depends upon: MAPREDUCE-4815 Speed up FileOutputCommitter#commitJob for many output files
• relates to: YARN-4602 Simple and Scalable Message Service for YARN application
• relates to: YARN-1489 [Umbrella] Work-preserving ApplicationMaster restart
• relates to: YARN-4758 Enable discovery of AMs by containers

Sub-Tasks

• YARN Registry based AM discovery with retry and in-flight task persistent via JHS: Patch Available
• Container Ids for an yarn application should be monotonically increasing in the scope of the application: Patch Available