[SPARK-45671] Implement an option similar to corrupt record column in State Data Source Reader

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 26/Oct/23 05:43
• Updated: 10/Nov/23 14:41

Description

Querying against the state would be most likely failing if the underlying state file is corrupted. There may be another case that the binary data (raw) state store read from state file does not fit with state schema and ends up with exception/fatal error in runtime.

(We can't catch the case where the data is loaded with incorrect schema if it does not throw an exception. We cannot add the schema for every data.)

To handle above cases without failure, we want to provide state rows for valid rows, with also providing binary data for corrupted rows (like we do for CSV/JSON) if users specify an option.

Issue Links

• depends upon: SPARK-45511 SPIP: State Data Source - Reader