[SPARK-25556] Predicate Pushdown for Nested fields

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: DB Tsai
• Reporter: DB Tsai
• Created: 27/Sep/18 18:00
• Updated: 01/Jul/20 01:34
• Resolved: 27/Mar/20 06:51

Description

This is an umbrella JIRA to support predicate pushdown for nested fields.

Issue Links

• is related to: SPARK-32142 Keep the original tests and codes to avoid potential conflicts in dev in ParquetFilterSuite and ParquetIOSuite
• links to: [Github] Pull Request #28955 (HyukjinKwon)
• links to: GitHub Pull Request #27728
• links to: GitHub Pull Request #28046

Sub-Tasks

• Benchmark Nested Parquet Predicate Pushdown: Resolved
• Parquet predicate pushdown for nested fields: Resolved
• ORC predicate pushdown for nested fields: Resolved
• Pushdown predicates for nested fields in DataSource Strategy: Resolved
• Just remove the unsupported predicates in Parquet: Resolved
• Refactor `DataSourceStrategy.scala` to minimize the changes to support nested predicate pushdown: Resolved
• Enable nested predicate pushdown per data sources: Resolved
• JSON predicate pushdown for nested fields: Open
• Avro predicate pushdown for nested fields: Open
• CSV predicate pushdown for nested fields: Open