[FLINK-22910] Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

We improved the ShuffleMaster interface by adding some lifecycle methods, including open, close, registerJob and unregisterJob. Besides, the ShuffleMaster now becomes a cluster level service which can be shared by multiple jobs. This is a breaking change to the pluggable shuffle service framework and the customized shuffle plugin needs to adapt to the new interface accordingly.

• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 08/Jun/21 04:03
• Updated: 28/Aug/21 13:09
• Resolved: 06/Aug/21 09:35

Description

The current ShuffleMaster has an unclear lifecycle which is inconsistent with the ShuffleEnvironment at the TM side. Besides, it is hard to Implement some important capabilities for remote shuffle service. For example, 1) release external resources when a job finished; 2) Stop or start tracking some partitions depending on the status of the external service or system.

We drafted a document[1] which proposed some simple changes to solve these issues. The document is still not wholly completed yet. We will start a discussion once it is finished.

 

[1] https://docs.google.com/document/d/1_cHoapNbx_fJ7ZNraSqw4ZK1hMRiWWJDITuSZrdMDDs/edit?usp=sharing

Issue Links

• is related to: FLINK-22672 Some enhancements for pluggable shuffle service framework
• links to: GitHub Pull Request #16465

Sub-Tasks

• Add lifecycle methods to ShuffleMaster: Closed
• Introduce ShuffleMasterContext to ShuffleMaster: Closed
• Provide JobID when apply shuffle resource by ShuffleMaster: Closed
• Make ShuffleMaster a cluster level shared service: Closed