[FLINK-36066] Introducing the AdaptiveGraphGenerator component

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Runtime / Coordination
• Assignee: Lei Yang
• Reporter: Junrui Li
• Created: 15/Aug/24 08:03
• Updated: 27/Aug/24 08:48

Description

To support the incremental generation of JobGraph and update the StreamGraph based on runtime information, we plan to introduce the AdaptiveGraphManager component, which will implement the AdaptiveGraphGenerator interface. The AdaptiveGraphGenerator interface is responsible for:

1. Generating the JobGraph
2. Responding to upstream job vertex finished events, generating JobVertex, and updating it to the JobGraph
3. Providing StreamGraphContext to allow StreamGraphOptimizer to add, delete, and modify StreamNode and StreamEdge