[SPARK-47960] Support Chaining Stateful Operators in TransformWithState

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Epic Link: Structured Streaming - Arbitrary State API v2
• Assignee: Bhuwan Sahni
• Reporter: Bhuwan Sahni
• Due: 26/Apr/24
• Created: 23/Apr/24 17:13
• Updated: 08/May/24 15:05
• Resolved: 08/May/24 00:20

Description

This issue tracks adding support to chain stateful operators after the Arbitrary State API, transformWithState. In order to support chaining, we need to allow the user to specify the new eventTimeColumn in the output from StatefulProcessor. Any watermark evaluation expressions downstream after transformWithState would use the user specified eventTimeColumn.

Issue Links

• links to: GitHub Pull Request #45376