[PARQUET-1542] Merge multiple I/O to one time I/O when read footer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: parquet-mr
• Labels: pull-request-available
• Assignee: Wang, Gang
• Reporter: Wang, Gang
• Created: 27/Feb/19 07:08
• Updated: 24/Oct/19 12:41
• Resolved: 24/Sep/19 05:00

Description

Currently in method readFooter of class ParquetFileReader, it will invoke method converter.readParquetMetadata. To decode file metadata,  there may be quite a number of times of I/O processing underlying, which may heavily slow down parquet read performance.

A better way is to read all the footer bytes in a single I/O, since we have already got footer length before, and pass down all the bytes to underlying.

 

And we have a test in our environment, this change may bring  50 percent improvement for some cases.

 

Issue Links

• links to: GitHub Pull Request #624