[FLINK-14060] Set slot sharing groups according to pipelined regions

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.10.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhu Zhu
• Reporter: Xintong Song
• Created: 12/Sep/19 02:51
• Updated: 05/Nov/19 10:25
• Resolved: 05/Nov/19 10:25

Description

StreamingJobGraphGenerator set slot sharing group for operators at compiling time.
Identify pipelined regions, with respect to allSourcesInSamePipelinedRegion
Set slot sharing groups according to pipelined regions 
By default, each pipelined region should go into a separate slot sharing group
If the user sets operators in multiple pipelined regions into same slot sharing group, it should be respected

The slot sharing group of a vertex is determined as:

• if a vertex slot sharing group is specified to be null, the slot sharing group is null
• if a vertex slot sharing group is specified to be a non-default value(the group name not "default"), it will be respected
• if a vertex slot sharing group is "default", it will be the slot sharing group of the vertex's logical pipelined region

The slot sharing group of a region is determined as:

• if allVerticesInSameSlotSharingGroupByDefault is true, all regions are in the same "default" slot sharing group
• if allVerticesInSameSlotSharingGroupByDefault is false, all regions will be in distinct slot sharing groups

This step should not introduce any behavior changes, given that later scheduled pipelined regions can reuse slots from previous scheduled pipelined regions. 

Issue Links

• blocks: FLINK-14062 Set managed memory fractions according to slot sharing groups
• blocks: FLINK-14314 Allocate shared slot resources respecting the resources of all vertices in the group
• is blocked by: FLINK-14312 Support building logical pipelined regions from JobGraph
• is blocked by: FLINK-14059 Introduce option allVerticesInSameSlotSharingGroupByDefault in ExecutionConfig
• links to: GitHub Pull Request #10007