Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-13450

Kafka broker to return alternating sets of listeners to clients for multipathing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • network
    • None

    Description

      It is not uncommon to deploy Kafka for high throughput use cases. In this context operators could have multiple network paths for client to broker traffic.

      The use of multiple network paths is not only to cover for failure conditions but also to balance load between the available paths. Common link speeds (currently) in data centers/cloud are 10Gbps/40Gbps and when there is a need to scale past the link speed multiple paths (multipathing) to the target (Kafka) is desired. The option of bundling multiple links in to a single channel is not commonly offered as a user facing feature in the Cloud.

      However when using DNS to load balance between the available network paths, TTL caching creates traffic skew between the network paths causing either deterioration or failure. 

      It would be ideal for the broker to return alternating sets of listener names to the metadata requests made by the client for even distribution of traffic between the available network paths. KIP 302 provides the ability for the clients to attempt to use all IP address for a name but does not cover situations when there is a need for multiple names each pointing to a separate network path. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            sudarshan@confluent.io Sudarshan Pathalam
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: