[IMPALA-7265] Cache remote file handles

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.1.0
• Fix Version/s: Impala 3.2.0
• Component/s: Backend

Docs Text

This introduced a new parameter cache_remote_file_handles, which modifies the behavior of the file handle cache. I think some pieces of documentation will need updates:
http://impala.apache.org/docs/build/html/topics/impala_scalability.html (section: "Scalability Considerations for NameNode Traffic with File Handle Caching")

• Target Version: Impala 3.2.0
• Epic Color: ghx-label-4
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 09/Jul/18 17:37
• Updated: 05/Oct/20 16:00
• Resolved: 07/Feb/19 04:55

Description

The file handle cache currently does not allow caching remote file handles. This means that clusters that have a lot of remote reads can suffer from overloading the NameNode. Impala should be able to cache remote file handles.

There are some open questions about remote file handles and whether they behave differently from local file handles. In particular:

1. Is there any resource constraint on the number of remote file handles open? (e.g. do they maintain a network connection?)
2. Are there any semantic differences in how remote file handles behave when files are deleted, overwritten, or appended?
3. Are there any extra failure cases for remote file handles? (i.e. if a machine goes down or a remote file handle is left open for an extended period of time)

The form of caching will depend on the answers, but at the very least, it should be possible to cache a remote file handle at the level of a query so that a Parquet file with multiple columns can share file handles.

Issue Links

• is related to: IMPALA-9485 Enable file handle cache for EC files
• is related to: IMPALA-10214 Ozone support for file handle cache
• is related to: IMPALA-8428 Add support for caching file handles on s3
• relates to: IMPALA-10202 Enable file handle cache for ABFS files

Sub-Tasks

• Impala Doc: Document remote file handle cache: Closed