[SPARK-37970] Introduce a new interface on streaming data source to notify the latest seen offset

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 20/Jan/22 11:03
• Updated: 11/Feb/22 07:07
• Resolved: 11/Feb/22 07:07

Description

We figure out the case of streaming data source that knowing the latest seen offset when restarting query would be handy and useful to implement some feature. One useful case is enabling the data source to track the offset by itself, for the case where the external storage of data source is not exposing any API to provide the latest available offset.

We will propose a new interface on streaming data source, which indicates Spark to give the latest seen offset whenever the query is being restarted. For the first start of the query, the initial offset of the data source should be retrieved from calling initialOffset.

Issue Links

• links to: [Github] Pull Request #35259 (HeartSaVioR)