[FLINK-11811] Implement etcd based RunningJobsRegistry and SubmittedJobGraphStore

• Type: Sub-task
• Status: Closed
• Priority: Minor
• Resolution: Abandoned
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: MalcolmSanders
• Created: 04/Mar/19 13:27
• Updated: 12/Oct/20 07:21
• Resolved: 12/Oct/20 07:21

Description

Zookeeper based RunningJobsRegistry uses zookeeper as a key-value store to keep the latest job statuses using persistent zookeeper nodes. So etcd based RunningJobsRegistry can use io.etcd.jetcd.KV to achieve the same goal.

Zookeeper based SubmittedJobGraphStore applies ZooKeeperStateHandleStore to store submitted each job graph in a dfs file while records its metadata, such as dfs location, to a zookeeper node in a zookeeper directory. etcd based SubmittedJobGraphStore will use EtcdSubmittedJobGraphStore to fulfill the target.

In addition, SubmittedJobGraphStore needs to watch this zookeeper directory to get notifications once znodes in such directory is created or deleted. Previous etcd API io.etcd.jetcd.Watch can watch a key as prefix by setting io.etcd.jetcd.options.WatchOption#withPrefix() which is just like watching a directory in zookeeper.

Issue Links

• is superceded by: FLINK-12884 FLIP-144: Native Kubernetes HA Service