[FLINK-21326] Optimize building topology when initializing ExecutionGraph

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Zhilong Hong
• Reporter: Zhilong Hong
• Created: 09/Feb/21 06:39
• Updated: 05/Mar/21 18:45
• Resolved: 05/Mar/21 18:37

Description

The main idea of optimizing the procedure of building topology is to put all the vertices that consumed the same result partitions into one group, and put all the result partitions that have the same consumer vertices into one consumer group. The corresponding data structure is ConsumedPartitionGroup and ConsumerVertexGroup. EdgeManager is used to store the relationship between the groups. The procedure of creating ExecutionEdge is replaced with building EdgeManager.

With these improvements, the complexity of building topology in ExecutionGraph decreases from O(N^2) to O(N). 

Furthermore, ExecutionEdge and all its related calls are replaced with ConsumedPartitionGroup and ConsumerVertexGroup.

The detailed design doc is located at: https://docs.google.com/document/d/1OjGAyJ9Z6KsxcMtBHr6vbbrwP9xye7CdCtrLvf8dFYw/edit?usp=sharing

Issue Links

• links to: GitHub Pull Request #14868