Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.0
-
None
Description
Currently, Structured Streaming directly puts the UnsafeRow into StateStore without any schema validation. It's a dangerous behavior when users reusing the checkpoint file during migration. Any changes or bug fix related to the aggregate function may cause random exceptions, even the wrong answer, e.g SPARK-28067.
Here we introduce an UnsafeRow format validation for the state store.
Attachments
Issue Links
- is blocked by
-
SPARK-31990 Streaming's state store compatibility is broken
- Resolved
- is related to
-
SPARK-31905 Add compatibility tests for streaming state store format
- Resolved
- links to