[SPARK-22364] unix_timestamp function sets valid dates to null

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.2.0
• Component/s: SQL

Environment

Windows 10, United Kingdom

• Assignee: Unassigned
• Reporter: Matthew Sinton-Hewitt
• Created: 26/Oct/17 14:05
• Updated: 26/Oct/17 17:31
• Resolved: 26/Oct/17 17:31

Description

org.apache.spark.sql.functions.unix_timestamp sets some valid dates to null.

The dates happen to be the at the start of Daylight Savings time (UK and possibly elsewhere).

val spark = SparkSession.builder.getOrCreate()

import spark.implicits._

spark.sparkContext.parallelize(
   Seq("25/03/2012 00:59", "25/03/2012 01:00", "25/03/2012 01:59", "25/03/2012 02:01"))
      .toDF("date")
      .select(unix_timestamp($"date", "dd/MM/yyyy HH:mm"))
      .show(false)

// results:
// 1332637140, null, null, 1332637260