[SPARK-36109] Fix flaky KafkaSourceStressSuite

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.1.2, 3.2.0
• Fix Version/s: 3.2.0, 3.1.3, 3.0.4
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 13/Jul/21 01:28
• Updated: 13/Jul/21 21:29
• Resolved: 13/Jul/21 08:23

Description

The test logic in KafkaSourceStressSuite is not stable. For example, https://github.com/apache/spark/runs/3049244904.

Once we add data to a topic and then delete the topic before checking data, the expected answer is different to retrieved data from the sink.

 

 

Issue Links

• links to: [Github] Pull Request #33311 (viirya)
• links to: [Github] Pull Request #33326 (viirya)
• links to: [Github] Pull Request #33326 (viirya)
• links to: [Github] Pull Request #33327 (viirya)
• links to: [Github] Pull Request #33327 (viirya)