[IMPALA-4623] Parquet Scanner - reduce NN RPC

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: ramp-up
• Epic Link: selective query optimizations
• Assignee: Joe McDonnell
• Reporter: Alan Choi
• Created: 08/Dec/16 01:31
• Updated: 28/Feb/18 00:24
• Resolved: 30/May/17 17:23

Description

The current Parquet scanner implementation treats a column in a row group as a "scan range". When reading a "scan range", Impala will issue a fopen RPC to the name node. Therefore, Impala will issue one RPC per column per row group. NN has a limited processing rate of fopen RPC and this can be a limiting factor on the query performance.

Fundamentally, there is no need to issue a fopen for each column. Impala should issue at most one fopen for each row group.

The current workaround of using file handle cache is not practical due to the large (1k byte) memory footprint per file handle cache. File handle cannot be shared by concurrent readers. So, if we have 10 queries reading the same file at the same time, we need 10 file handles cached.

Issue Links

• breaks: IMPALA-5857 Crash in impala::DiskIoMgr::ScanRange::Close-> tc_free while running concurrent TPC-DS