[SPARK-31174] unix_timestamp() function returning NULL values for corner cases (daylight saving)

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 2.3.4
• Component/s: Spark Shell, SQL
• Assignee: Unassigned
• Reporter: Piotr Skąpski
• Created: 17/Mar/20 16:04
• Updated: 12/Dec/22 18:10
• Resolved: 23/Mar/20 05:06

Description

Running below code does not return 4 values due to one timestamp not being correctly created (possible problem with daylight saving):

spark.sql("""SELECT from_unixtime(unix_timestamp('2020-03-08 01:00:00'), 'yyyyMMdd') t1, from_unixtime(unix_timestamp('2020-03-08 02:00:00'), 'yyyyMMdd') t2, from_unixtime(unix_timestamp('2020-03-08 03:00:00'), 'yyyyMMdd') t3, from_unixtime(unix_timestamp('2020-03-08 04:00:00'), 'yyyyMMdd') t4""").show

+--------+----+--------+--------+
|      t1|  t2|      t3|      t4|
+--------+----+--------+--------+
|20200308|null|20200308|20200308|
+--------+----+--------+--------+

This unexpected NULL value caused us problems as we did not expect it.