Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7958

Transactions are broken with kubernetes hosted brokers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.0
    • 2.1.1
    • None
    • None
    • cp-kakfka 2.1.1-1, kafka-streams 2.1.1

    Description

      After a rolling re-start in a kubernetes-like environment, brokers may change IP address.  From our logs it seems that the transaction manager in the brokers never re-resolves the DNS name of other brokers, keeping stale pod IPs.  Thus transactions stop working.  

      [2019-02-20 02:20:20,085] WARN [TransactionCoordinator id=1001] Connection to node 0 (khaki-joey-kafka-0.khaki-joey-kafka-headless.hyperspace-dev/10.233.124.181:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

      [2019-02-20 02:20:57,205] WARN [TransactionCoordinator id=1001] Connection to node 1 (khaki-joey-kafka-1.khaki-joey-kafka-headless.hyperspace-dev/10.233.122.67:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

      This is from the log from broker 1001 which was restarted first, followed by 1 and then 0.  The log entries are from the day after the rolling restart.

      I note a similar issue was fixed for clients 2.1.1  https://issues.apache.org/jira/browse/KAFKA-7755.  We are using streams lib 2.1.1

      Update We are now testing with Kafka 2.1.1 (docker cp-kafka 5.1.2-1) and it looks like this may be resolved.  Would be good to get confirmation though.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              t-d-d Thomas Dickinson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: