[FLINK-17598] Implement FileSystemHAServices for native K8s setups

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Component/s: Deployment / Kubernetes, Runtime / Coordination
• Assignee: Unassigned
• Reporter: Canbin Zheng
• Created: 09/May/20 12:17
• Updated: 07/Apr/21 09:56
• Resolved: 07/Apr/21 09:56

Description

At the moment we use Zookeeper as a distributed coordinator for implementing JobManager high availability services. But in the cloud-native environment, there is a trend that more and more users prefer to use Kubernetes as the underlying scheduler backend while Storage Object as the Storage medium, both of these two services don't require Zookeeper deployment.

As a result, in the K8s setups, people have to deploy and maintain their Zookeeper clusters for solving JobManager SPOF. This ticket proposes to provide a simplified FileSystem HA implementation with the leader-election removed, which saves the efforts of Zookeeper deployment.

To achieve this, we plan to

1. Introduce a FileSystemHaServices which implements the HighAvailabilityServices.
2. Replace Deployment with StatefulSet to ensure at most one semantics, preventing potential concurrent access to the underlying FileSystem.

Issue Links

• is related to: FLINK-12751 Create file based HA support