[SPARK-47331] Serialization using case classes/primitives/POJO based on SQL encoder for Arbitrary State API v2.

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Epic Link: Structured Streaming - Arbitrary State API v2
• Assignee: Jing Zhan
• Reporter: Jing Zhan
• Created: 08/Mar/24 23:38
• Updated: 14/Mar/24 21:30
• Resolved: 11/Mar/24 00:21

Description

In the new operator for arbitrary state-v2, we cannot rely on the session/encoder being available since the initialization for the various state instances happens on the executors. Also, we can only support limited state types with the available encoders. Hence, for the state serialization, we propose to serialize primitives/case classes/POJO with SQL encoder. Leveraging SQL encoder can speed up the serialization.

Issue Links

• links to: GitHub Pull Request #45447