[FLINK-11105] Add a new implementation of the HighAvailabilityServices using etcd

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Component/s: Runtime / Coordination
• Assignee: Unassigned
• Reporter: Yang Wang
• Created: 07/Dec/18 16:18
• Updated: 12/Oct/20 07:19
• Resolved: 12/Oct/20 07:18

Description

In flink, we use HighAvailabilityServices to do many things, e.g. RM/JM leader election and retrieval. ZooKeeperHaServices is an implementation of HighAvailabilityServices using Apache ZooKeeper. It is very easy to integrate with hadoop ecosystem. However, the cloud native and micro service are become more and more popular. We just need to follow the step and add a new implementation EtcdHaService using etcd.

Now flink has supported to run StandaloneSession on kubernetes and FLINK-9953 start to make an native integration with kubernetes. If we have the EtcdHaService, both of them will benefit from this and we will not have deploy a zookeeper service on kubernetes cluster.

Issue Links

• is related to: FLINK-2366 HA Without ZooKeeper
• is superceded by: FLINK-12884 FLIP-144: Native Kubernetes HA Service
• links to: JobManager High Availability using etcd Proposal

Sub-Tasks

• Import shaded jetcd: Closed
• Implement etcd based LeaderElectionService and LeaderRetrievalService: Closed
• Implement etcd based SharedCount or sequencer: Closed
• Implement etcd based StateHandleStore: Closed
• Implement etcd based CheckpointRecoveryFactory: Closed
• Implement etcd based RunningJobsRegistry and SubmittedJobGraphStore: Closed
• Composite etcd based HighAvailability components: Closed