[FLINK-29665] Support flexible subpartition range divisions

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.17.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Lijie Wang
• Reporter: Lijie Wang
• Created: 17/Oct/22 11:48
• Updated: 29/Dec/22 13:02
• Resolved: 29/Dec/22 13:02

Description

Currently, the subpartition range division algorithm is fixed (data-independent), and the TaskDeploymentDescriptor(TDD) can be generated according to the number of subpartitions and downstream parallelism. In order to support dividing according to the amount of data in the subpartition range, the scheduler needs to support flexible subpartition range divisions. The scheduler will be responsible for deciding the subpartition range division infos for each job vertex, and then pass the division infos to job vertices when initializing them.

Issue Links

• links to: GitHub Pull Request #21162