[SPARK-41498] Union does not propagate Metadata output

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2, 3.2.0, 3.1.3, 3.2.1, 3.3.0, 3.2.2, 3.3.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Fredrik Klauß
• Created: 12/Dec/22 12:41
• Updated: 14/Mar/23 15:39

Description

Currently, the Union operator does not propagate any metadata output. This makes it impossible to access any metadata if a Union operator is used, even though the children have the exact same metadata output.
Example:

 

val df1 = spark.read.load(path1)
val df2 = spark.read.load(path2)
df1.union(df2).select("_metadata.file_path"). // <-- fails

Issue Links

• links to: [Github] Pull Request #38941 (fred-db)
• links to: [Github] Pull Request #40371 (cloud-fan)