[SPARK-34392] Invalid ID for offset-based ZoneId since Spark 3.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1
• Fix Version/s: 3.0.3, 3.1.2, 3.2.0
• Component/s: SQL
• Assignee: karl wang
• Reporter: Yuming Wang
• Created: 07/Feb/21 08:31
• Updated: 12/Dec/22 18:11
• Resolved: 26/Feb/21 17:04

Description

How to reproduce this issue:

select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");

Spark 2.4:

spark-sql> select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");
2020-02-07 08:00:00
Time taken: 0.089 seconds, Fetched 1 row(s)

Spark 3.x:

spark-sql> select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00");
21/02/07 01:24:32 ERROR SparkSQLDriver: Failed in [select to_utc_timestamp("2020-02-07 16:00:00", "GMT+8:00")]
java.time.DateTimeException: Invalid ID for offset-based ZoneId: GMT+8:00
	at java.time.ZoneId.ofWithPrefix(ZoneId.java:437)
	at java.time.ZoneId.of(ZoneId.java:407)
	at java.time.ZoneId.of(ZoneId.java:359)
	at java.time.ZoneId.of(ZoneId.java:315)
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.getZoneId(DateTimeUtils.scala:53)
	at org.apache.spark.sql.catalyst.util.DateTimeUtils$.toUTCTime(DateTimeUtils.scala:814)

Issue Links

• links to: [Github] Pull Request #31624 (Karl-WangSK)
• links to: [Github] Pull Request #31755 (maropu)