[SPARK-48772] State Data Source Read Change Feed

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Epic Link: Fine-grained State Data Source
• Assignee: Yuchen Liu
• Reporter: Yuchen Liu
• Created: 01/Jul/24 23:14
• Updated: 10/Jul/24 07:17
• Resolved: 10/Jul/24 07:17

Description

The current state reader can only return the entire state at a specific version. If an error occurs related to state, knowing the change of state across versions to find out at which version state starts to go wrong is important for debugging purposes. This PR adds ability of showing the evolution of state as Change Data Capture (CDC) format to state data source.

An example usage:

.format("statestore")
.option("readChangeFeed", true)
.option("changeStartBatchId", 5) #required 
.option("changeEndBatchId", 10)  #not required, default: latest batch Id available

Issue Links

• links to: GitHub Pull Request #47188