[SPARK-26812] PushProjectionThroughUnion nullability issue

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.2, 2.1.3, 2.2.3, 2.3.4, 2.4.0
• Fix Version/s: 2.4.4, 3.0.0
• Component/s: SQL
• Labels: correctness
• Assignee: Marco Gaido
• Reporter: Bogdan Raducanu
• Created: 02/Feb/19 01:43
• Updated: 06/Sep/21 08:49
• Resolved: 01/Apr/19 14:24

Description

Union output data types are the output data types of the first child.
However the other union children may have different values nullability.
This means that we can't always push down a project on the children.

To reproduce

Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1")
sql("SELECT 1 AS b").write.saveAsTable("table2")
sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL SELECT a FROM table1")
 sql("select * from test1").show

This fails becaus the plan is no longer resolved.
The plan is broken by the PushProjectionThroughUnion rule which pushed down a cast to map<string,string> with values nullability=true on a child with type map<string, string> with values nullability=false.

Issue Links

• is duplicated by: SPARK-27685 `union` doesn't promote non-nullable columns of struct to nullable
• relates to: SPARK-36673 Incorrect Unions of struct with mismatched field name case
• links to: GitHub Pull Request #23726
• links to: GitHub Pull Request #24604