[SPARK-26655] Support multiple aggregates in Structured Streaming append mode

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Arun Mahadevan
• Created: 17/Jan/19 21:16
• Updated: 06/May/22 04:17

Description

Right now multiple aggregates are not supported in structured streaming.

However, in append mode, the aggregates are emitted only after the watermark passes the threshold (e.g. the window boundary) and the emitted value is not affected by further late data. So it possible to chain multiple aggregates in 'Append' output mode without worrying about retractions.

However the current event time watermarks in structured streaming are tracked at a global level and this does not work when aggregates are chained.

We need to track the watermarks at individual operator level so that each operator can make progress independently and not rely on global min or max value.

Issue Links

• links to: GitHub Pull Request #23576