[HIVE-25827] Parquet file footer is read multiple times, when multiple splits are created in same file

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-2
• Component/s: Iceberg integration
• Labels: performance pull-request-available
• Assignee: Ádám Szita
• Reporter: Rajesh Balamohan
• Created: 20/Dec/21 21:50
• Updated: 16/Nov/22 13:50
• Resolved: 22/Jun/22 09:08

Description

With large files, it is possible that multiple splits are created in the same file. With current codebase, "ParquetRecordReaderBase" ends up reading file footer for each split. 

It can be optimized not to read footer information multiple times for the same file.

 

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedParquetRecordReader.java#L160

 

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ParquetRecordReaderBase.java#L91

 

 

 

Issue Links

• relates to: HADOOP-18028 High performance S3A input stream with prefetching & caching
• links to: GitHub Pull Request #3368