[FLINK-25329] Improvement of execution graph store in flink session cluster for jobs

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0, 1.12.5, 1.13.3
• Fix Version/s: 1.15.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Fang Yong
• Reporter: Fang Yong
• Created: 15/Dec/21 11:57
• Updated: 09/Feb/22 08:06
• Resolved: 09/Feb/22 08:06

Description

Flink session cluster uses files to store info of jobs after they reach termination with `FileExecutionGraphInfoStore`, each job will generate one file. When the cluster executes many small jobs concurrently, there will be many disk related operations, which will

1> Increase the CPU usage of `Dispatcher`
2> Decrease the performance of the jobs in the cluster.

We hope to improve the disk operations in `FileExecutionGraphInfoStore` to increase the performance of session cluster, or support memory store.

Issue Links

• links to: GitHub Pull Request #18360