[SPARK-26345] Parquet support Column indexes

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 12/Dec/18 09:08
• Updated: 13/Jan/23 07:06
• Resolved: 31/Jan/21 02:06

Description

Parquet 1.11 supports column indexing. Spark can supports this feature for better read performance.

More details:

https://issues.apache.org/jira/browse/PARQUET-1201

 

Benchmark result:

https://github.com/apache/spark/pull/31393#issuecomment-769767724

This feature is enabled by default, and users can disable it by setting parquet.filter.columnindex.enabled to false.

Issue Links

• relates to: SPARK-34859 Vectorized parquet reader needs synchronization among pages for column index