[SPARK-34889] Introduce MergingSessionsIterator merging elements directly which belong to the same session

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 29/Mar/21 03:16
• Updated: 23/Jun/21 20:05
• Resolved: 23/Jun/21 20:05

Description

This issue tracks effort on introducing MergingSessionsIterator, which enables to merge elements belong to the same session directly. This would be quite performant compared to UpdatingSessionIterator. Note that MergingSessionsIterator can only apply to the cases aggregation can be applied altogether, so there're still rooms for UpdatingSessionIterator to be used.

This issue also introduces MergingSessionsExec which is the physical node on leveraging MergingSessionsIterator to sort the input rows and aggregate rows according to the session windows.

Issue Links

• links to: [Github] Pull Request #31987 (HeartSaVioR)
• links to: [Github] Pull Request #31987 (HeartSaVioR)