[BEAM-2534] KafkaIO should allow gaps in message offsets

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: io-java-kafka
• Assignee: Raghu Angadi
• Reporter: Raghu Angadi
• Created: 28/Jun/17 19:32
• Updated: 16/May/20 13:17
• Resolved: 10/Jul/17 19:56

Description

KafkaIO reader logs a warning when it notices gaps in offsets for messages. While such gaps are not expected for normal Kafka topics, there could be gaps when log compaction is enabled (which deletes older messages for a key).

This warning log is not very useful. Also we should take such gaps while estimating backlog.

Issue Links

• links to: GitHub Pull Request #3461
• links to: GitHub Pull Request #3514