[KAFKA-3120] Consumer doesn't get messages from some partitions after reassign

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.9.0.0
• Environment: Linux
• Assignee: Unassigned
• Reporter: Jakub Neubauer
• Created: 19/Jan/16 19:52
• Updated: 24/May/16 16:46
• Resolved: 24/May/16 16:46

Description

I tested some scenario on one-node cluster with this setup:

• A topic with 5 partitions, 1 replica.
• One producer (new java client)
• 2 consumers were started (let's say A and B). Using the new java client. 2 partitions to A, 3 partitions to B were assigned.

Then I stopped one of the consumers (cleanly). The partitions were re-assigned (The consumer got all 5 partitions in the "ConsumerRebalanceListener.onPartitionsAssigned" listener.

But as messages were produced, the living consumer received only messages of some partitions (magically those that belonged to the now-dead consumer).
The messages were not lost. After I restarted the second consumer, it got the messages that it previously didn't get. But without restarting, the messages were not consumed by it.

It is quite serious issue, since there is no sign of something being wrong. Everything seems to be working. So the administrator has no chance to get the idea that (only some) messages are not consumed on the "healthy" system.

Issue Links

• duplicates: KAFKA-2877 Messages sometimes not delivered by new consumer after Kafka restart