[SPARK-33306] TimezoneID is needed when there cast from Date to String

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1, 3.1.0
• Fix Version/s: 3.0.2, 3.1.0
• Component/s: SQL
• Assignee: EdisonWang
• Reporter: EdisonWang
• Created: 31/Oct/20 01:46
• Updated: 22/Feb/21 00:56
• Resolved: 31/Oct/20 22:15

Description

A simple way to reproduce this is 

spark-shell --conf spark.sql.legacy.typeCoercion.datetimeToString.enabled

scala> sql("""

select a.d1 from
 (select to_date(concat('2000-01-0', id)) as d1 from range(1, 2)) a
 join
 (select concat('2000-01-0', id) as d2 from range(1, 2)) b
 on a.d1 = b.d2

""").show

 

it will throw

java.util.NoSuchElementException: None.get
 at scala.None$.get(Option.scala:529)
 at scala.None$.get(Option.scala:527)
 at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:56)
 at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:56)
 at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId$lzycompute(Cast.scala:253)
 at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId(Cast.scala:253)
 at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter$lzycompute(Cast.scala:287)
 at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter(Cast.scala:287)

Issue Links

• duplicates: SPARK-33420 BroadCastJoin failure when keys on join side has cast from DateTyte to String
• links to: [Github] Pull Request #30213 (WangGuangxin)
• links to: [Github] Pull Request #30213 (WangGuangxin)
• links to: [Github] Pull Request #30223 (MaxGekk)
• links to: [Github] Pull Request #30223 (MaxGekk)