Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-6598

Kafka to support using ETCD beside Zookeeper

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: clients, core
    • Labels:
      None

      Description

      The current Kafka implementation is bound to Zookeeper to store its metadata for forming a cluster of nodes (producer/consumer/broker).
      As Kafka is becoming popular for streaming in various environments where Zookeeper is either not easy to deploy/manage or there are better alternatives to it there is a need
      to run Kafka with other metastore implementation than Zookeeper.

      etcd can provide the same semantics as Zookeeper for Kafka and since etcd is the favorable choice in certain environments (e.g. Kubernetes) Kafka should be able to run with etcd.
      From the user's point of view should be straightforward to configure to use etcd by just simply specifying a connection string that point to etcd cluster.

      To avoid introducing instability the original interfaces should be kept and only the low level Zookeeper API calls should be replaced with etcd API calls in case Kafka is configured
      to use etcd.

      On the long run (which is out of scope of this jira) there should be an abstract layer in Kafka which then various metastore implementations would implement.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stoader Sebastian Toader
            • Votes:
              14 Vote for this issue
              Watchers:
              34 Start watching this issue

              Dates

              • Created:
                Updated: