[KAFKA-10791] Kafka Metadata older epoch problem

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0
• Component/s: clients
• Environment: Kubernetes cluster,
• Assignee: Unassigned
• Reporter: KRISHNA SARVEPALLI
• Created: 01/Dec/20 16:25
• Updated: 02/Dec/20 13:45

Description

We are using Kafka in production with 5 brokers and 3 zookeepers. We are running Kafka and zookeeper in Kubernetes and storage is managed by PVC using NFS. We are using topic with 60 partitions.

The cluster was running successfully from almost 50 days since the last restart. Last week (11/28) two brokers were down. Team is still researching for the root cause of broker failures. 

Since we are using K8s the brokers came back up immediately (in less than 5minutes). But we have issue on the producer applications and consumer applications while downloading the metadata. Please check the attached images.

We have enabled debug logs for one of the applications and it seems like Kafka brokers are returning metadata with leader_epoch value of 0 where as in the client Metadata cache it was maintained at 6 for most of the partitions. 

Eventually we are forced to restart all the producer apps (around 35-40 micro services) and they are all able to download the metadata since it's first time didn't face any issue and was able to produce the messages.

As part of troubleshooting, we have checked the zookeeper key/value pairs registered by Kafka and we can see that leader_epoch was set back to 0 for almost all partitions. And we have checked for another topic which is used by other apps, their leader_epoch was in good shape and ctime and mtime are also updated correctly. Please check the attached screenshots.

Please refer the stackoverflow issue that we have reported:

https://stackoverflow.com/questions/65055299/kafka-producer-not-able-to-download-refresh-metadata-after-brokers-were-restar

 

Broker Configs:

--override zookeeper.connect=zookeeper:2181
--override advertised.listeners=PLAINTEXT://kafka,SASL_SSL://kafka
--override log.dirs=/opt/kafka/data/logs
--override broker.id=kafka
--override num.network.threads=3
--override num.io.threads=8
--override default.replication.factor=3
--override auto.create.topics.enable=true
--override delete.topic.enable=true
--override socket.send.buffer.bytes=102400
--override socket.receive.buffer.bytes=102400
--override socket.request.max.bytes=104857600
--override num.partitions=30
--override num.recovery.threads.per.data.dir=1
--override offsets.topic.replication.factor=3
--override transaction.state.log.replication.factor=3
--override transaction.state.log.min.isr=1
--override log.retention.hours=48
--override log.segment.bytes=1073741824
--override log.retention.check.interval.ms=300000
--override zookeeper.connection.timeout.ms=6000
--override confluent.support.metrics.enable=true
--override group.initial.rebalance.delay.ms=0
--override confluent.support.customer.id=anonymous
--override ssl.truststore.location=kafka.broker.truststore.jks
--override ssl.truststore.password=changeit
--override ssl.keystore.location=kafka.broker.keystore.jks
--override ssl.keystore.password=changeit
--override ssl.keystore.type=PKCS12
--override ssl.key.password=changeit
--override listeners=SASL_SSL://0.0.0.0:9093,PLAINTEXT://0.0.0.0:9092
--override authorizer_class_name=kafka.security.auth.SimpleAclAuthorizer
--override ssl.endpoint.identification.algorithm
--override ssl.client.auth=requested
--override sasl.enabled.mechanisms=SCRAM-SHA-512
--override sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
--override security.inter.broker.protocol=SASL_SSL
--override super.users=test:test
--override zookeeper.set.acl=false