[SPARK-45939] SPIP: Structured Streaming - Arbitrary State API v2

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Component/s: Structured Streaming
• Epic Link: Structured Streaming - Arbitrary State API v2
• Assignee: Unassigned
• Reporter: Anish Shrigondekar
• Shepherd: Jungtaek Lim
• Created: 15/Nov/23 20:23
• Updated: 26/Jul/24 19:07
• Resolved: 26/Jul/24 19:07

Description

SPIP: Structured Streaming - Arbitrary State API v2

 

We are planning to introduce a new operator for Spark Structured Streaming around arbitrary stateful operations. The operator(s) we have today (mapGroupsWithState/flatMapGroupsWithState) have a bunch of limitations (as noted in Q3 below) that prevent the use of the APIs for complex/advanced use-cases for stateful stream processing. We plan to remove these limitations with the introduction of a new API to handle state manipulation for streaming queries.

 

As part of this effort, we want to address:

• composite types

• multiple state variables (flexible data modeling)

• state TTL

• callback/timer improvements

• schema evolution

• initial state

• side outputs

• output modes

• co-process functions

• chaining operators after new operator etc