[SPARK-26345] Parquet support Column indexes

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 12/Dec/18 09:08
• Updated: 13/Jan/23 07:06
• Resolved: 31/Jan/21 02:06

Description

Parquet 1.11 supports column indexing. Spark can supports this feature for better read performance.

More details:

https://issues.apache.org/jira/browse/PARQUET-1201

 

Benchmark result:

https://github.com/apache/spark/pull/31393#issuecomment-769767724

This feature is enabled by default, and users can disable it by setting parquet.filter.columnindex.enabled to false.

Issue Links

• relates to: SPARK-34859 Vectorized parquet reader needs synchronization among pages for column index

Sub-Tasks

• Upgrade built-in Hive to 2.3.8: Resolved
• Upgrade Jackson to 2.11.4: Resolved
• Upgrade parquet to 1.11.1: Resolved
• Upgrade to Avro 1.10.1: Resolved
• Do not push down partition filters to ParquetScan for DataSourceV2: Resolved
• Vectorized reader support column index: Resolved