[KAFKA-7070] KafkaConsumer#committed might unexpectedly shift consumer offset

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: clients
• Assignee: Unassigned
• Reporter: Jan Lukavský
• Created: 18/Jun/18 16:02
• Updated: 18/Jun/18 20:35

Description

When client uses manual partition assignment (e.g. KafkaConsumer#assign), but then accidentally calls KafkaConsumer#committed (for whatever reason, most probably bug in user code), then the offset gets shifted to latest, possibly skipping any unconsumed messages, or producing duplicates. The reason is that the call to KafkaConsumer#committed invokes AbstractCoordinator, which tries to fetch committed offset, but doesn't find group.id (will be probably even empty). This might cause Fetcher to receive invalid offset for partition and reset it to the latest offset.

Although this is primarily bug in user code, it is very hard to track it down. The call to KafkaConsumer#committed might probably throw exception when called on client without auto partition assignment.