[SPARK-31885] Incorrect filtering of old millis timestamp in parquet

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.1.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Apache Spark
• Reporter: Max Gekk
• Created: 01/Jun/20 10:45
• Updated: 01/Jun/20 18:23
• Resolved: 01/Jun/20 15:14

Description

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.1.0-SNAPSHOT
      /_/

Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_242)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
scala> spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "CORRECTED")
scala> Seq(java.sql.Timestamp.valueOf("1000-06-14 08:28:53.123")).toDF("ts").write.mode("overwrite").parquet("/Users/maximgekk/tmp/ts_millis_old_filter")

scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").show(false)
+-----------------------+
|ts                     |
+-----------------------+
|1000-06-14 08:28:53.123|
+-----------------------+


scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").filter($"ts" === "1000-06-14 08:28:53.123")
res6: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [ts: timestamp]

scala> spark.read.parquet("/Users/maximgekk/tmp/ts_millis_old_filter").filter($"ts" === "1000-06-14 08:28:53.123").show(false)
+---+
|ts |
+---+
+---+

Issue Links

• links to: [Github] Pull Request #28693 (MaxGekk)
• links to: [Github] Pull Request #28699 (MaxGekk)
• links to: [Github] Pull Request #28699 (MaxGekk)