[SPARK-49863] NormalizeFloatingNumbers degrades nullability of expressions with nested structs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Nikhil Sheoran
• Reporter: Nikhil Sheoran
• Created: 03/Oct/24 00:58
• Updated: 09/Oct/24 03:28
• Resolved: 09/Oct/24 03:28

Description

For an AttributeReference where the dataType is a nested struct such that the internal struct requires normalization (has a floating type expression), we end up not correctly propagating the nullability of an expression.

For example, for an expression like:

namedStruct("struct", namedStruct("double", <DoubleType-field>) 

The dataType prior to normalization is:

StructType(StructField("struct", StructType(StructField("double", DoubleType, true, {})), false, {}))

whereas post-normalization, the dataType becomes:

StructType(StructField("struct", StructType(StructField("double", DoubleType, true, {})), true, {})) 

 

We ended up converting the `nullable` attribute of the "double" field from `false` to `true`.

Issue Links

• links to: GitHub Pull Request #48331