Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
1.9.0, 1.10.0, 1.9.1
Description
This was discovered in Spark, see SPARK-26677. From the Spark PR:
// Repeat the values to get dictionary encoding. Seq(Some("A"), Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/foo") spark.read.parquet("/tmp/foo").where("NOT (value <=> 'A')").show() +-----+ |value| +-----+ +-----+
// Use plain encoding. Seq(Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/bar") spark.read.parquet("/tmp/bar").where("NOT (value <=> 'A')").show() +-----+ |value| +-----+ | null| +-----+
This is a correctness issue.
Attachments
Issue Links
- blocks
-
PARQUET-1512 Release Parquet Java 1.10.1
-
- Resolved
-
- causes
-
SPARK-26677 Incorrect results of not(eqNullSafe) when data read from Parquet file
-
- Resolved
-
- links to