[SPARK-5465] Data source version of Parquet doesn't push down And filters properly

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1
• Fix Version/s: 1.3.0
• Component/s: SQL
• Target Version/s: 1.2.2, 1.3.0
• Assignee: Unassigned
• Reporter: Cheng Lian
• Created: 28/Jan/15 21:48
• Updated: 02/Feb/15 02:52
• Resolved: 02/Feb/15 02:52

Description

The current implementation combines all predicates and then tries to convert it to a single Parquet filter predicate. In this way, the Parquet filter predicate can not be generated if any component of the original filters can not be converted. (code lines here).

For example, a > 10 AND a < 20 can be successfully converted, while a > 10 AND a < b can't because Parquet doesn't accept filters like a < b.

Issue Links

• links to: [Github] Pull Request #4255 (liancheng)