[SPARK-47734] Fix flaky pyspark.sql.dataframe.DataFrame.writeStream doctest by stopping streaming query

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0, 3.5.2, 3.4.3
• Component/s: PySpark, Tests
• Labels: pull-request-available
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 04/Apr/24 23:52
• Updated: 08/Apr/24 21:27
• Resolved: 05/Apr/24 02:15

Description

https://issues.apache.org/jira/browse/SPARK-47199 didn't fix the flakiness in the pyspark.sql.dataframe.DataFrame.writeStream doctest : the problem is not that we are colliding on the test but, rather, that the test is starting a background thread to write to a directory then deleting that directory from the main test thread, something which is inherently race prone.

The fix is simple: stop the streaming query in the doctest itself, similar to other streaming doctest examples.

Issue Links

• relates to: SPARK-47199 Add prefix into TemporaryDirectory to avoid flakiness
• links to: GitHub Pull Request #45885
• links to: GitHub Pull Request #45908