[SPARK-23477] Misleading exception message when union fails due to metadata

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 2.2.1
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Tomasz Bartczak
• Created: 21/Feb/18 10:52
• Updated: 12/Dec/22 18:10
• Resolved: 03/Mar/18 09:23

Description

When I have two DF's that are different only in terms of metadata in fields inside a struct - I cannot union them but the error message shows that they are the same:

df = spark.createDataFrame([{'a':1}])
a = df.select(struct('a').alias('x'))
b = df.select(col('a').alias('a',metadata={'description':'xxx'})).select(struct(col('a')).alias('x'))
a.union(b).printSchema()

gives:

An error occurred while calling o1076.union.
: org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. struct<a:bigint> <> struct<a:bigint> at the first column of the second table

and this part:

struct<a:bigint> <> struct<a:bigint>

does not make any sense because those are the same.

 

Since metadata must be the same for union -> it should be incuded in the error message