[SAMZA-1481] KafkaSystemConsumer does not properly track highwatermark in some cases.

• Type: Bug
• Status: Open
• Priority: Trivial
• Resolution: Unresolved
• Fix Version/s: 0.15.0
• Assignee: Jake Maes
• Reporter: Jake Maes
• Created: 01/Nov/17 20:29
• Updated: 11/Apr/18 00:10

Description

The KafkaSystemConsumer currently only updates the isAtHighWatermark when it receives a payload from Kafka.

This does not work when:
1. The topic is empty
2. The oldest or newest offset changes on the broker
3. An invalid initial offset is used for the consumer and the Kafka auto reset policy kicks in.

The isAtHighWatermark flag is crucial for topics that we bootstrap (checkpoint, changelog, coordinator) because we poll infinitely until we're at head of stream. And if this flag is wrong, we poll indefinitely.

Issue Links

• depends upon: SAMZA-1644 Blocking poll on KafkaConsumer can miss the last batch of messages