[KAFKA-14366] Kafka consumer rebalance issue, offsets points back to very old committed offset

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.1
• Component/s: consumer, offset manager
• Environment: Production

Language

Java

• Assignee: Unassigned
• Reporter: Chetan
• Created: 08/Nov/22 07:18
• Updated: 02/Dec/22 06:03

Description

Hi All,

We are facing an issue while the client consumer restart (again not all restarts are ending up with this issue) and during the re-balancing scenario, sometimes one of the partition offsets goes back a long way from the committed offset.

Scenario :

Assume we have 4 instances of consumer and restarts of consumer one after the other.

1. At the time of starting restarts assume the offset on partition 10 of a topic being consumed is pointing to 50000. (last offset of the topic and 0 lag)
2. When restarts start (rebalancing) suddenly the offsets start pointing to 20000.
3. While all the restarts are going on the consumer who is attached starts reading from 20000 and goes on.
4. Once all rebalance is completed, and all messages from 20000 to 50000 offset has been read (where it had stopped initially)
   We end up having around 30K duplicates.

(The numbers here are just an example, in production, we are facing huge duplicates and every two rebalance during restarts of consumer out of 10 restart exercise activity ends up in such duplicates and not all partitions and only one or two partitions behave this way and randomly)

This seems to be a bug. I am attaching all screenshots for reference as well.

Can someone kindly help out here?