[SPARK-9935] EqualNullSafe not processed in OrcRelation

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: SQL
• Target Version/s: 1.6.0
• External issue ID: https://issues.apache.org/jira/browse/SPARK-9814
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 13/Aug/15 10:03
• Updated: 12/Dec/22 17:50
• Resolved: 13/Aug/15 23:07

Description

it is an issue followed by SPARK-9814.

Now datasources (after selectFilters() in org.apache.spark.sql.execution.datasources.DataSourceStrategy) can pass EqualNotNull to relations but OrcRelation does not take and process this.

(ParquetRelation has also the same issue but it seems Parquet does not support the null-safe equal filter. It is confusing whether Parquet should support this or treat this as EqualFilter with additional codes in Spark code. Therefore, I made a separate issue only for ORC)

Issue Links

• links to: [Github] Pull Request #8163 (HyukjinKwon)