[SPARK-33420] BroadCastJoin failure when keys on join side has cast from DateTyte to String

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.1
• Component/s: SQL

Environment

spark 3.0.1 hadoop 2.9.2 

• Assignee: Unassigned
• Reporter: qinyu
• Created: 11/Nov/20 08:39
• Updated: 22/Feb/21 00:56
• Resolved: 22/Feb/21 00:56

Description

when use spark as below : 

spark.sql(
""" create table table1(a1 INT , a2 STRING)

using parquet

""".stripMargin).show()

spark.sql(
""" create table table2(b1 INT , b2 STRING)

using parquet

""".stripMargin).show()

spark.sql(
""" select /*+ BROADCAST(a) */ * from table1 a join table2 b

on cast(to_date(a.a2) as string) = b.b2

""".stripMargin).show()

Exception following will be thrown : 

 java.util.NoSuchElementException: None.getjava.util.NoSuchElementException: None.get at scala.None$.get(Option.scala:529) at scala.None$.get(Option.scala:527) at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId(datetimeExpressions.scala:56) at org.apache.spark.sql.catalyst.expressions.TimeZoneAwareExpression.zoneId$(datetimeExpressions.scala:56) at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId$lzycompute(Cast.scala:253) at org.apache.spark.sql.catalyst.expressions.CastBase.zoneId(Cast.scala:253) at org.apache.spark.sql.catalyst.expressions.CastBase.dateFormatter$lzycompute(Cast.scala:287)

 

Issue Links

• is duplicated by: SPARK-33306 TimezoneID is needed when there cast from Date to String
• links to: [Github] Pull Request #30329 (hzyangkai)