[SPARK-31894] Introduce UnsafeRow format validation for streaming state store

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Yuanjian Li
• Reporter: Yuanjian Li
• Created: 02/Jun/20 18:22
• Updated: 31/Jul/20 09:24
• Resolved: 19/Jun/20 05:57

Description

Currently, Structured Streaming directly puts the UnsafeRow into StateStore without any schema validation. It's a dangerous behavior when users reusing the checkpoint file during migration. Any changes or bug fix related to the aggregate function may cause random exceptions, even the wrong answer, e.g SPARK-28067.

Here we introduce an UnsafeRow format validation for the state store.

Issue Links

• is blocked by: SPARK-31990 Streaming's state store compatibility is broken
• is related to: SPARK-31905 Add compatibility tests for streaming state store format
• links to: [Github] Pull Request #28707 (xuanyuanking)
• links to: [Github] Pull Request #29315 (xuanyuanking)