[KAFKA-7698] Kafka Broker fail to start: ProducerFencedException thrown from producerstatemanager.scala!checkProducerEpoch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Labels: easyfix
• Assignee: Unassigned
• Reporter: Ming Liu
• Reviewer: Jason Gustafson
• Created: 03/Dec/18 18:06
• Updated: 24/Feb/23 20:02
• Resolved: 24/Feb/23 20:02

Description

During our operation of Kafka, we frequently saw this failure: 

   There was an error in one of the threads during logs loading: org.apache.kafka.common.errors.ProducerFencedException:

[06:57:09,697] INFO [ProducerStateManager partition=interaction_events-127] Loading producer state from snapshot file '/data/disk5/kafka/interaction_events-127/00000000092130764817.snapshot' (kafka.log.ProducerStateManager)
[06:57:09,698] INFO [Log partition=interaction_events-127, dir=/data/disk5/kafka] Completed load of log with 11 segments, log start offset 91975003024 and log end offset 92130764817 in 12701 ms (kafka.log.Log)
[06:57:09,701] ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is no longer valid. There is probably another producer with a newer epoch. 63 (request epoch), 66 (server epoch) (kafka.log.LogManager)
[06:57:09,705] INFO [ProducerStateManager partition=client_interaction_events_authorid_enrichment-20] Writing producer snapshot at offset 92418754384 (kafka.log.ProducerStateManager)
[06:57:09,707] ERROR [KafkaServer id=2] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is no longer valid. There is probably another producer with a newer epoch. 63 (request epoch), 66 (server epoch)
{code:java}
 

Issue Links

• is caused by: KAFKA-9144 Early expiration of producer state can cause coordinator epoch to regress
• links to: GitHub Pull Request #5992