[SPARK-48589] Add option snapshotStartBatchId and snapshotPartitionId to state data source

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Epic Link: Fine-grained State Data Source
• Assignee: Yuchen Liu
• Reporter: Yuchen Liu
• Created: 11/Jun/24 20:14
• Updated: 02/Jul/24 20:16
• Resolved: 02/Jul/24 20:16

Description

Define two new options, snapshotStartBatchId and snapshotPartitionId, for the existing state reader. Both of them should be provided at the same time.

1. When there is no snapshot file at that batch (note there is an off-by-one issue between version and batch Id), throw an exception.
2. Otherwise, the reader should continue to rebuild the state by reading delta files only, and ignore all snapshot files afterwards.
3. Note that if a batchId option is already specified. That batchId is the ending batchId, we should then end at that batchId.

Issue Links

• links to: GitHub Pull Request #46944