[SPARK-9264] When dealing with Union/Intersect/Except, we cast FloatType/DoubleType to the wrong DecimalType

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.4.2, 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Davies Liu
• Reporter: Yin Huai
• Created: 22/Jul/15 22:22
• Updated: 24/Jul/15 07:22
• Resolved: 24/Jul/15 07:22

Description

The problem is https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/HiveTypeCoercion.scala#L361-L362. When we union/intersect/except, for a column, if the data type of it in one table is a fixed precision decimal type and that in another table is a FloatType/DoubleType, we cast FloatType to Decimal(7, 7) and cast DoubleType to Decimal(15, 15), respectively. It is wrong.

I tried the following in 1.4

sqlContext.sql("select a from (select cast(200.101 as double) as a union all select cast(200.101 as decimal(6,3)) as a) tmp").show

I got

+-------------------+
|                  a|
+-------------------+
|               null|
|200.101000000000000|
+-------------------+

Issue Links

• links to: [Github] Pull Request #7605 (davies)