[SPARK-11621] ORC filter pushdown not working properly after new unhandled filter interface.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.6.0
• Component/s: SQL
• External issue ID: SPARK-10978
• Assignee: Unassigned
• Reporter: Hyukjin Kwon
• Created: 10/Nov/15 06:05
• Updated: 12/Dec/22 18:10
• Resolved: 09/Dec/15 15:12

Description

After the new interface to get rid of filters predicate-push-downed which are already processed in datasource-level (https://github.com/apache/spark/pull/9399), it dose not push down filters for ORC.

This is because at DataSourceStrategy, all the filters are treated as unhandled filters.

Also, since ORC does not support to filter fully record by record but instead rough results came out, the filters for ORC should not go to unhandled filters.

Issue Links

• duplicates: SPARK-11661 We should still pushdown filters returned by a data source's unhandledFilters
• relates to: SPARK-11661 We should still pushdown filters returned by a data source's unhandledFilters