Details
-
Epic
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.0.0
-
None
-
Fine-grained State Data Source
Description
The current state reader API replays the state store rows from the latest snapshot and newer delta files if any. The issue with this mechanism is that sometimes, the snapshot files could be wrongly constructed, or user want to know the change of state across batches. We need to improve the State Reader so that it can handle a variety of fine-grained requirements. For example, reconstruct a state based on arbitrary snapshot; support CDC mode for state evolution.
Attachments
Attachments
Issue Links
- links to
Issues in epic
|
SPARK-48589 | Add option snapshotStartBatchId and snapshotPartitionId to state data source | Resolved | Yuchen Liu | ||
|
SPARK-48772 | State Data Source Read Change Feed | Resolved | Yuchen Liu | ||
|
SPARK-48850 | Add documentation for new options added to State Data Source | Resolved | Yuchen Liu |
SPARK-48588
Fine-grained State Data Source
false
SPARK-48588
Fine-grained State Data Source