[SPARK-38227] Apply strict nullability of nested column in time window / session window

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.1, 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 16/Feb/22 07:29
• Updated: 21/Feb/22 05:45
• Resolved: 21/Feb/22 05:45

Description

In TimeWindow and SessionWindow, we define dataType of these function expressions as StructType having two nested columns "start" and "end", which is "nullable".

And we replace these expressions in the analyzer via corresponding rules, TimeWindowing for TimeWindow, and SessionWindowing for SessionWindow.

The rules replace the function expressions with Alias, referring CreateNamedStruct. For the value side of CreateNamedStruct, we don't specify anything about nullability, which leads to a risk the value side may be interpreted (or optimized) as non-nullable, which would make inconsistency.

We should make sure the nullability of columns in CreateNamedStruct remains the same with dataType definition on these function expressions.

Issue Links

• links to: [Github] Pull Request #35543 (HeartSaVioR)