[SPARK-39585] Multiple Stateful Operators in Structured Streaming

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Boyang Jerry Peng
• Created: 24/Jun/22 20:50
• Updated: 12/Dec/22 18:11

Description

Currently, Structured Streaming only supports one stateful operator per pipeline.  This constraint excludes Structured Streaming to be used for many use cases such ones involving chained time window aggregations, chained stream-stream outer equality join, stream-stream time interval join followed by time window aggregation.  Enabling multiple stateful operators within a pipeline in Structured Streaming will open up many additional use cases for its usage as well as be on par with other stream processing engines.

Issue Links

• Is contained by: SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark