[SPARK-42388] Avoid unnecessary parquet footer reads when no filters in vectorized reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Mars
• Reporter: Mars
• Created: 09/Feb/23 07:20
• Updated: 01/Aug/24 12:08
• Resolved: 16/Apr/23 04:10

Description

Parquet footer is now read twice even if there are no filters requiring pushdown in vectorized parquet reader.
When the NameNode is under high pressure, it will cost time to read twice. Actually we can avoid this unnecessary parquet footer reads and use footer metadata in VectorizedParquetRecordReader.

Issue Links

• causes: SPARK-48950 Corrupt data from parquet scans
• relates to: SPARK-48571 Reduce the number of accesses to S3 object storage
• links to: [Github] Pull Request #39950 (yabola)
• links to: GitHub Pull Request #39950