[FLINK-4344] Implement new JobManager

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Runtime / Coordination
• Labels: flip-6
• Assignee: Unassigned
• Reporter: Stephan Ewen
• Created: 09/Aug/16 22:20
• Updated: 28/Feb/19 14:13
• Resolved: 13/Jul/18 07:14

Description

This is the parent issue for the efforts to implement the JobManager changes based on FLIP-6 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)

Because of the breadth of changes, we should implement a new version of the JobManager (let's call it JobMaster) rather than updating the current JobManager. That will allow us to keep a working master branch.
At the point when the new cluster management is on par with the current implementation, we will drop the old JobManager and rename the JobMaster to JobManager.

Here is a sketch about the new design of the JobManager.

Issue Links

• blocks: FLINK-4319 Rework Cluster Management (FLIP-6)
• depends upon: FLINK-4457 Make the ExecutionGraph independent of Akka
• is blocked by: FLINK-7871 SlotPool should release its unused slot to RM
• is blocked by: FLINK-7870 SlotPool should cancel the slot request to RM if not need any more.
• is blocked by: FLINK-4459 Introduce SlotProvider for Scheduler
• is broken by: FLINK-6434 There may be allocatedSlots leak in SlotPool
• requires: FLINK-4568 Replace ActorGateway in WebRuntimeMonitor

Sub-Tasks

• Introduce rpc protocols implemented by job manager: Closed
• Leadership Election among JobManagers: Resolved
• Implement job master registration at resource manager: Resolved
• Use Listeners to monitor execution status: Closed
• Submit Job and setup ExecutionGraph: Resolved
• Implement job master skeleton for managing single job: Closed
• interaction between new JobManager and new ResourceManager: Resolved
• new JobManager handle cluster components failure: Resolved
• JobManager handle TaskManager's registration: Resolved
• new JobManager HA: Resolved
• Prevent BLOCKING partition data loss when task is finished: Resolved
• Implement a simple slot provider for the new job manager: Resolved
• Replace SlotAllocationFuture with flink's own future: Resolved
• Migrate some job execution related akka messages to rpc calls: Resolved
• JobManager handle TaskManager's slot offering: Resolved
• Harden JobMaster logic: Resolved
• Retry connection in case of a ResourceManager heartbeat timeout: Resolved
• Set JobMaster leader session id in main thread: Closed
• Fence JobMaster: Closed
• Add support for scheduling with slot sharing: Closed