[SPARK-45045] SPARK-43183 broke various tests in 3rd party streaming data sources

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 01/Sep/23 06:40
• Updated: 04/Sep/23 02:42
• Resolved: 04/Sep/23 02:42

Description

SPARK-43183 made a change regarding behavior to the StreamingQueryListener as well as StreamingQuery API, while the intention was more about introducing the change in the former one.

I just got some reports from 3rd party data sources that they encountered innocent test failures during upgrade to 3.5.0 (RC), and it looks like they are mostly concerned about the behavioral change for StreamingQuery API. (It's lot less convenient for developers to rely on streaming query listener for testing.)

That said, to avoid 3rd party ecosystem be bugging with new behavior, we should probably revert the behavior for StreamingQuery API at least.