Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-12884

FLIP-144: Native Kubernetes HA Service

    XMLWordPrintableJSON

Details

    Description

      Currently flink only supports HighAvailabilityService using zookeeper. As a result, it requires a zookeeper cluster to be deployed on k8s cluster if our customers needs high availability for flink. If we support HighAvailabilityService based on native k8s APIs, it will save the efforts of zookeeper deployment as well as the resources used by zookeeper cluster. It might be especially helpful for customers who run small-scale k8s clusters so that flink HighAvailabilityService may not cause too much overhead on k8s clusters.

      Previously FLINK-11105 has proposed a HighAvailabilityService using etcd. As NathanHowell suggested in FLINK-11105, since k8s doesn't expose its own etcd cluster by design (see Securing etcd clusters), it also requires the deployment of etcd cluster if flink uses etcd to achieve HA.

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. FLINK-17709 Active Kubernetes integration phase 3 - Advanced Features

          • Major - Major loss of function.
          • Closed

          Technical Debt - FLINK-19545 Add e2e test for native Kubernetes HA

          • Major - Major loss of function.
          • Closed
          supercedes

          Sub-task - The sub-task of the issue FLINK-11806 Import shaded jetcd

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11807 Implement etcd based LeaderElectionService and LeaderRetrievalService

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11808 Implement etcd based SharedCount or sequencer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11809 Implement etcd based StateHandleStore

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11810 Implement etcd based CheckpointRecoveryFactory

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11811 Implement etcd based RunningJobsRegistry and SubmittedJobGraphStore

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Sub-task - The sub-task of the issue FLINK-11812 Composite etcd based HighAvailability components

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. FLINK-11105 Add a new implementation of the HighAvailabilityServices using etcd

          • Major - Major loss of function.
          • Closed
          links to

          Web Link FLIP-144 Native Kubernetes HA for Flink

          mentioned in

          Page Loading...

          Page Loading...

          There are no Sub-Tasks for this issue.

          Activity

            People

              wangyang0918 Yang Wang
              ssy MalcolmSanders
              Votes:
              4 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: