[FLINK-4356] new JobManager HA

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Later
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Jing Zhang
• Created: 10/Aug/16 09:23
• Updated: 19/Nov/21 15:01
• Resolved: 29/Jun/17 12:46

Description

1. for standalone mode, LocalDispatcher watch JobMaster
LocalDispatcher detect the failure of JobMaster, recover jobGraph and Libraries from persistent storage, spawn a new JobManager
new JobMaster compete for leadership, save address to zookeeper storage
new JobMaster registers at ResourceManager
new JobMaster recover Execution of its job (execution graph) from latest completed checkpoint
2. for yarn mode, YarnApplicationMasterRunner create a ProcessReaper of JobMaster
ProcessReaper monitor JobMaster, kill JVM upon JobMaster termination
Yarn will create a new AppMaster which contains a new JobManager, JobGraph and Libraries are retrieved as startup artifacts
new JobMaster compete for leadership, save address to zookeeper storage
new JobMaster registers at ResourceManager
new JobMaster recover Execution of its job (execution graph) from latest completed checkpoint