[SPARK-31227] Non-nullable null type should not coerce to nullable type

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 23/Mar/20 11:13
• Updated: 12/Dec/22 17:50
• Resolved: 26/Mar/20 08:52

Description

scala> spark.range(10).selectExpr("array()").printSchema()
root
 |-- array(): array (nullable = false)
 |    |-- element: null (containsNull = false)


scala> spark.range(10).selectExpr("concat(array()) as arr").printSchema()
root
 |-- arr: array (nullable = false)
 |    |-- element: null (containsNull = false)


scala> spark.range(10).selectExpr("concat(array(), array(1)) as arr").printSchema()
root
 |-- arr: array (nullable = false)
 |    |-- element: integer (containsNull = true)

The last case should not coerce to nullable type.

Issue Links

• links to: GitHub Pull Request #27991