[PARQUET-2415] Reuse hadoop file status and footer in ParquetRecordReader

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: parquet-hadoop
• Assignee: Unassigned
• Reporter: Wan Kun
• Created: 21/Dec/23 01:32
• Updated: 23/Jun/24 03:33

Description

DESCRIPTION

Spark will send a listStatus RPC to get hadoop file status and read the parquet file footer before reading the parquet file. And send a same listStatus RPC to get the same hadoop file status and read the footer again in ParquetRecordReader. We can reuse the file status and the footer.

PLANS

Save the hadoop file status in the ParquetMetadata and save the ParquetMetadata in the input split, so we can reuse them when init a new ParquetRecordReader.

Issue Links

• links to: GitHub Pull Request #1242