Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10791

Kafka Metadata older epoch problem

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None
    • Environment:
      Kubernetes cluster,

      Description

      We are using Kafka in production with 5 brokers and 3 zookeepers. We are running Kafka and zookeeper in Kubernetes and storage is managed by PVC using NFS. We are using topic with 60 partitions.

      The cluster was running successfully from almost 50 days since the last restart. Last week (11/28) two brokers were down. Team is still researching for the root cause of broker failures. 

      Since we are using K8s the brokers came back up immediately (in less than 5minutes). But we have issue on the producer applications and consumer applications while downloading the metadata. Please check the attached images.

      We have enabled debug logs for one of the applications and it seems like Kafka brokers are returning metadata with leader_epoch value of 0 where as in the client Metadata cache it was maintained at 6 for most of the partitions. 

      Eventually we are forced to restart all the producer apps (around 35-40 micro services) and they are all able to download the metadata since it's first time didn't face any issue and was able to produce the messages.

      As part of troubleshooting, we have checked the zookeeper key/value pairs registered by Kafka and we can see that leader_epoch was set back to 0 for almost all partitions. And we have checked for another topic which is used by other apps, their leader_epoch was in good shape and ctime and mtime are also updated correctly. Please check the attached screenshots.

      Please refer the stackoverflow issue that we have reported:

      https://stackoverflow.com/questions/65055299/kafka-producer-not-able-to-download-refresh-metadata-after-brokers-were-restar

       

      Broker Configs:

      --override zookeeper.connect=zookeeper:2181
      --override advertised.listeners=PLAINTEXT://kafka,SASL_SSL://kafka
      --override log.dirs=/opt/kafka/data/logs
      --override broker.id=kafka
      --override num.network.threads=3
      --override num.io.threads=8
      --override default.replication.factor=3
      --override auto.create.topics.enable=true
      --override delete.topic.enable=true
      --override socket.send.buffer.bytes=102400
      --override socket.receive.buffer.bytes=102400
      --override socket.request.max.bytes=104857600
      --override num.partitions=30
      --override num.recovery.threads.per.data.dir=1
      --override offsets.topic.replication.factor=3
      --override transaction.state.log.replication.factor=3
      --override transaction.state.log.min.isr=1
      --override log.retention.hours=48
      --override log.segment.bytes=1073741824
      --override log.retention.check.interval.ms=300000
      --override zookeeper.connection.timeout.ms=6000
      --override confluent.support.metrics.enable=true
      --override group.initial.rebalance.delay.ms=0
      --override confluent.support.customer.id=anonymous
      --override ssl.truststore.location=kafka.broker.truststore.jks
      --override ssl.truststore.password=changeit
      --override ssl.keystore.location=kafka.broker.keystore.jks
      --override ssl.keystore.password=changeit
      --override ssl.keystore.type=PKCS12
      --override ssl.key.password=changeit
      --override listeners=SASL_SSL://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
      --override authorizer_class_name=kafka.security.auth.SimpleAclAuthorizer
      --override ssl.endpoint.identification.algorithm
      --override ssl.client.auth=requested
      --override sasl.enabled.mechanisms=SCRAM-SHA-512
      --override sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
      --override security.inter.broker.protocol=SASL_SSL
      --override super.users=test:test
      --override zookeeper.set.acl=false

       

        Attachments

        1. zookeeper-state.png
          299 kB
          KRISHNA SARVEPALLI
        2. zookeeper-leader-epoch.png
          300 kB
          KRISHNA SARVEPALLI
        3. Topic-Recreated.png
          293 kB
          KRISHNA SARVEPALLI
        4. Kafka-Client-Issue.png
          943 kB
          KRISHNA SARVEPALLI

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              chkrishna411 KRISHNA SARVEPALLI

              Dates

              • Created:
                Updated:

                Issue deployment