[SPARK-46722] Add a test regarding to backward compatibility check for StreamingQueryListener in Spark Connect (Scala/PySpark)

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.5.0, 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect, Structured Streaming
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 15/Jan/24 09:19
• Updated: 07/Apr/24 22:02
• Resolved: 16/Jan/24 03:23

Description

When we added onQueryIdle to StreamingQueryListener, we missed to add backward compatibility test(s) into StreamingQueryListenerSuite.

We added the compatibility test in pyspark when we fixed an issue on pyspark specific via https://issues.apache.org/jira/browse/SPARK-45631, but we didn't add the symmetric test in Spark Connect (both Scala and PySpark). We should add the test.

Issue Links

• links to: GitHub Pull Request #44736
• links to: GitHub Pull Request #45913