[FLINK-19543] Implement RunningJobsRegistry, JobGraphStore based on Kubernetes API

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: Deployment / Kubernetes, Runtime / Coordination
• Labels: pull-request-available
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 09/Oct/20 03:07
• Updated: 25/Feb/22 11:00
• Resolved: 07/Nov/20 17:02

Description

• RunningJobsRegistry
• Registry for the running jobs. All the jobs in the registry will be recovered when JobManager failover. 

• JobGraphStore
• JobGraph instances for running JobManagers. Note that only the meta information(aka location reference, DFS path) will be stored in the Zookeeper/ConfigMap. The real data is stored on the DFS.

Each component(Dispatcher, ResourceManager, JobManager, RestEndpoint) will have a dedicated ConfigMap. All the HA information relevant for a specific component will be stored in a single ConfigMap. So the Dispatcher's ConfigMap would then contain the current leader, the running jobs and the pointers to the persisted JobGraphs.

Issue Links

• causes: FLINK-26285 ZooKeeperStateHandleStore does not handle not existing nodes properly in getAllAndLock
• links to: GitHub Pull Request #13864
• links to: GitHub Pull Request #13864