[FLINK-4343] Implement new TaskManager

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.0
• Component/s: Runtime / Coordination
• Labels: flip-6
• Assignee: Till Rohrmann
• Reporter: Stephan Ewen
• Created: 09/Aug/16 22:03
• Updated: 28/Feb/19 14:13
• Resolved: 13/Jul/18 06:33

Description

This is the parent issue for the efforts to implement the TaskManager changes based on FLIP-6 (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077)

Because of the breadth of changes, we should implement a new version of the TaskManager (let's call it TaskExecutor) rather than updating the current TaskManager. That will allow us to keep a working master branch.

At the point when the new cluster management is on par with the current implementation, we will drop the old TaskManager and rename the TaskExecutor to TaskManager.

Issue Links

• blocks: FLINK-4319 Rework Cluster Management (FLIP-6)
• depends upon: FLINK-4424 Make network environment start-up/shutdown independent of JobManager association
• depends upon: FLINK-4455 Replace ActorGateways in NetworkEnvironment by interfaces
• depends upon: FLINK-4456 Replace ActorGateway in Task by interface
• links to: GitHub Pull Request #2640
• links to: TaskManager design document

Sub-Tasks

• Implement TaskManager basic startup of all components in java: Closed
• Implement TaskManagerRunner to construct related components for TaskManager: Closed
• Implement TaskManager side of registration at ResourceManager: Closed
• Implement heartbeat logic between TaskManager and ResourceManager: Closed
• Implement TaskManager's SlotAllocationTable: Closed
• Implement TaskManager side of slot allocation from ResourceManager: Closed
• Implement heartbeat logic between TaskManager and JobManager: Closed
• Implement TaskManager registration slot at JobManager: Closed
• Implement TaskManager deployment task into slot: Closed
• Port TaskManager logic to TaskExecutor: Closed
• Replace TaskManagerRuntimeInfo by TaskManagerConfiguration: Closed
• Send slot report to RM when registering: Closed
• Retry JobManager/ResourceManager connection in case of timeout: Closed
• Add entry point for TaskManagerRunner: Closed
• Send SlotReport as part of the heartbeat payload to the RM: Closed
• Add proper command line parsing tool to TaskManagerRunner.main: Closed
• Handle slot requests occuring before RM registration completes: Resolved