[SPARK-25559] Just remove the unsupported predicates in Parquet

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 27/Sep/18 18:17
• Updated: 23/Apr/20 20:36
• Resolved: 29/Sep/18 00:47

Description

Currently, in ParquetFilters, if one of the children predicates is not supported by Parquet, the entire predicates will be thrown away. In fact, if the unsupported predicate is in the top level And condition or in the child before hitting Not or Or condition, it's safe to just remove the unsupported one as unhandled filters.

Issue Links

• is depended upon by: SPARK-31537 Backport SPARK-25559 Remove the unsupported predicates in Parquet when possible
• links to: [Github] Pull Request #22574 (dbtsai)
• links to: [Github] Pull Request #22574 (dbtsai)
• links to: [Github] Pull Request #22679 (gatorsmile)
• links to: [Github] Pull Request #22679 (gatorsmile)