[FLINK-22674] Provide JobID when apply shuffle resource by ShuffleMaster

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.14.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Jin Xing
• Created: 17/May/21 03:10
• Updated: 05/Aug/21 02:47
• Resolved: 05/Aug/21 02:45

Description

In current Flink 'pluggable shuffle service' framework, only PartitionDescriptor and ProducerDescriptor are included as parameters in ShuffleMaster#registerPartitionWithProducer.

But when extending a remote shuffle service based on 'pluggable shuffle service', JobID is also needed when apply shuffle resource from remote cluster. It can be used as an identification to link shuffle resource with the corresponding job:

1. Remote shuffle cluster can isolate or do capacity control on shuffle resource between jobs;
2. Remote shuffle cluster can use JobID for shuffle data cleanup when job is lost thus to avoid file leak;

Issue Links

• links to: GitHub Pull Request #16453