[SPARK-48411] Add E2E test for DropDuplicateWithinWatermark

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, SS
• Labels: pull-request-available
• Assignee: Yuchen Liu
• Reporter: Wei Liu
• Created: 24/May/24 07:22
• Updated: 12/Jun/24 03:46
• Resolved: 12/Jun/24 03:46

Description

Currently we do not have a e2e test for DropDuplicateWithinWatermark, we should add one. We can simply use one of the test written in Scala here (with the testStream API) and replicate it to python:

https://github.com/apache/spark/commit/0e9e34c1bd9bd16ad5efca77ce2763eb950f3103

 

The change should happen in https://github.com/apache/spark/blob/eee179135ed21dbdd8b342d053c9eda849e2de77/python/pyspark/sql/tests/streaming/test_streaming.py#L29

 

so we can test it in both connect and non-connect.

 

Test with:

```
python/run-tests --testnames pyspark.sql.tests.streaming.test_streaming
python/run-tests --testnames pyspark.sql.tests.connect.streaming.test_parity_streaming
```

Issue Links

• links to: GitHub Pull Request #46740