[PARQUET-1510] Dictionary filter skips null values when evaluating not-equals.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.0, 1.10.0, 1.9.1
• Fix Version/s: 1.11.0, 1.10.1
• Component/s: parquet-mr
• Labels: correctness pull-request-available
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 25/Jan/19 16:43
• Updated: 23/Jun/24 03:30
• Resolved: 28/Jan/19 17:07

Description

This was discovered in Spark, see SPARK-26677. From the Spark PR:

// Repeat the values to get dictionary encoding.
Seq(Some("A"), Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/foo")
spark.read.parquet("/tmp/foo").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
+-----+

// Use plain encoding.
Seq(Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/bar")
spark.read.parquet("/tmp/bar").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
| null|
+-----+

This is a correctness issue.

Issue Links

• blocks: PARQUET-1512 Release Parquet Java 1.10.1
• causes: SPARK-26677 Incorrect results of not(eqNullSafe) when data read from Parquet file
• links to: GitHub Pull Request #603