[SPARK-25214] Kafka v2 source may return duplicated records when `failOnDataLoss` is `false`

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Labels: correctness
• Target Version/s: 2.4.0
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 23/Aug/18 17:25
• Updated: 24/Aug/18 21:45
• Resolved: 24/Aug/18 20:58

Description

When there are missing offsets, Kafka v2 source may return duplicated records when failOnDataLoss=false because it doesn't skip missing offsets.

Issue Links

• links to: [Github] Pull Request #22207 (zsxwing)
• links to: [Github] Pull Request #22230 (zsxwing)