[SPARK-45178] Fallback to use single batch executor for Trigger.AvailableNow with unsupported sources rather than using wrapper

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 15/Sep/23 04:52
• Updated: 21/Sep/23 01:58
• Resolved: 20/Sep/23 02:05

Description

We have observed the case where wrapper implementation of Trigger.AvailableNow (
AvailableNowDataStreamWrapper and subclasses) is not fully compatible with 3rd party data source and brought up correctness issue.
 
While we could persuade 3rd party data source to support Trigger.AvailableNow, pursuing all 3rd parties to do this is too aggressive and challenging goal we never be able to make. Also, it may not be also possible to come up with the wrapper implementation which would have zero issue with any arbitrary source.
 
As a mitigation, we want to make a slight behavioral change for such case, falling back to single batch execution (a.k.a. Trigger.Once) rather than using wrapper implementation. The exact behavior between Trigger.AvailableNow and Trigger.Once are different so it's technically behavioral change, but it's probably lot less surprised than failing the query.
 
For extreme case where users are confident that there will be no issue at all on using wrapper, we will come up with a flag to provide the previous behavior.

Issue Links

• links to: GitHub Pull Request #42940