[SPARK-28489] KafkaOffsetRangeCalculator.getRanges may drop offsets

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3
• Fix Version/s: 2.4.4, 3.0.0
• Component/s: Structured Streaming
• Labels: correctness dataloss
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 24/Jul/19 00:09
• Updated: 25/Jun/20 18:22
• Resolved: 26/Jul/19 07:12

Description

KafkaOffsetRangeCalculator.getRanges may drop offsets due to round off errors.

 

This only affects queries using "minPartitions" option. A workaround is just removing the "minPartitions" option from the query.

Issue Links

• is caused by: SPARK-23541 Allow Kafka source to read data with greater parallelism than the number of topic-partitions
• links to: GitHub Pull Request #25237