[IMPALA-6403] Enable file handle reuse for multiple scan ranges within the same file for an HDFS Scan node

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Backend
• Epic Color: ghx-label-8
• Assignee: Joe McDonnell
• Reporter: Mostafa Mokhtar
• Created: 17/Jan/18 00:16
• Updated: 22/Dec/20 21:12

Description

Impala creates a file handle per scan range, for queries that read multiple columns per scan range un-necessarily large load is added to the HDFS NameNode which limits scalability on large clusters.

For a given set of scan ranges against a file within a Scan Node a single file handle should be created an reused to avoid excessive RPCs. 

Issue Links

• is duplicated by: IMPALA-6361 File handle cache should be shared across multiple IO threads
• relates to: IMPALA-5212 consider switching to pread by default