[SPARK-10816] EventTime based sessionization (session window)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Reynold Xin
• Created: 24/Sep/15 22:08
• Updated: 21/Jul/21 03:32
• Resolved: 21/Jul/21 01:58

Description

Currently structured streaming supports two kinds of windows: tumbling window and sliding window. Another useful window function is session window. Which is not supported by SS.

Unlike time window (tumbling window and sliding window), session window doesn't have static window begin and end time. Session window creation depends on defined session gap which can be static or dynamic.

For static session gap, the events which are falling in a certain period of time (gap) are considered as a session window. A session window closes when it does not receive events for the gap. For dynamic gap, the gap could be changed from event to event.

Issue Links

• is duplicated by: SPARK-34427 Session window support in SS
• is duplicated by: SPARK-22565 Session-based windowing
• links to: [Baidu] SPIP - Session Window Support For Structure Streaming
• links to: [Github] Pull Request #22482 (HeartSaVioR)
• links to: [Github] Pull Request #22482 (HeartSaVioR)
• links to: [Github] Pull Request #22583 (xuanyuanking)
• links to: [Github] Pull Request #31570 (viirya)
• links to: [Github] Pull Request #31570 (viirya)
• links to: [Github] Pull Request #31937 (HeartSaVioR)
• links to: [Github] Pull Request #31937 (HeartSaVioR)
• links to: [HWX] Analysis of implementation of session window: comparison patches between HWX and Baidu
• links to: [HWX] Detailed design of native support of session window
• links to: [HWX] SPIP - Support session window natively
• links to: GitHub Pull Request #22482
• links to: GitHub Pull Request #22583

Sub-Tasks

• Introduce UpdatingSessionIterator adjusting session information on elements: Resolved
• Introduce MergingSessionsIterator merging elements directly which belong to the same session: Resolved
• Introduce "prefix match scan" feature on state store: Resolved
• Introduce state store manager for session window in streaming query: Resolved
• Introduce MergingSortWithSessionWindowStateIterator sorting input rows and rows in state efficiently: Resolved
• Support native session window: Resolved
• Document session window in Structured Streaming guide doc: Resolved
• Dynamic gap duration in session window: Resolved