[PARQUET-1739] Make Spark SQL support Column indexes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.11.1
• Component/s: parquet-mr
• Assignee: Yuming Wang
• Reporter: Yuming Wang
• Created: 08/Jan/20 10:56
• Updated: 23/Jun/24 03:31
• Resolved: 30/Jan/21 07:03

Description

Make Spark SQL support Column indexes.

Issue Links

• blocks: SPARK-26346 Upgrade parquet to 1.11.1
• is related to: PARQUET-1201 Column indexes

Sub-Tasks

• Make ParquetFileReader.getFilteredRecordCount public: Resolved
• Some filters throws ArrayIndexOutOfBoundsException: Resolved
• No result for partition key included in Parquet file: Resolved
• Changed the data order after DataFrame reuse: Resolved