[KAFKA-13087] production environment throw huge OutOfOrderSequenceException("Invalid sequence number for new epoch at offset..") when enable idempotent function

• Type: Task
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: clients, core
• Assignee: Unassigned
• Reporter: huangfulibo
• Created: 14/Jul/21 11:18
• Updated: 14/Jul/21 11:18

Description

Hi Kafka team,

I am a new Kafka user.

some basic informations

Kafka server version is 2.6.0

Kafka client version  is 2.6.0

enable idempotent function and not use transaction function

 

After we upgrade Kafka version from 2.3.1 to 2.6.0, we find that in our product environment kafka server the ProducerStateManager class throw out huge numbers OutOfOrderSequenceException ("Invalid sequence number for new epoch at offset....") intermittently.

the details logs as bellow:
2021-07-10T03:35:31.790257-04:00 pc8bkafka04 kafka[2653]: [2021-07-10 03:35:31,790] ERROR [ReplicaManager broker=4] Error processing append operation on partition platform.foundation.user.sync-1 (kafka.server.ReplicaManager)
2021-07-10T03:35:31.790895-04:00 pc8bkafka04 kafka[2653]: org.apache.kafka.common.errors.OutOfOrderSequenceException: Invalid sequence number for new epoch at offset 7007971 in partition platform.foundation.user.sync-1: 6159 (request epoch), 1 (seq. number)
I have spent some days to investigate this issue, and have read some related Kafka client and server source code based on Kafka 2.6 version, but i can not reproduce the Exception in my local develop environment, and i can not find the root cause, it confused me some days.

 

cloud your Kafka team give me some guides to troubleshooting this issue.

 

 

Thanks,

huangfulibo