[SPARK-41145] Assert the offset range for file stream source in Trigger.AvailableNow

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 3.4.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 15/Nov/22 06:42
• Updated: 15/Nov/22 07:06
• Resolved: 15/Nov/22 07:06

Description

We encountered the issue where the data source did not properly implement the offset with Trigger.AvailableNow, and the query ran with processing same data continuously without stopping.

We would like to proactively avoid such case for most used data sources. I'll create a new JIRA ticket for Kafka data source as well.

Issue Links

• is cloned by: SPARK-41146 Assert the offset range for Kafka source in Trigger.AvailableNow