[IMPALA-8428] Add support for caching file handles on s3

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Backend
• Epic Color: ghx-label-6
• Assignee: Sahil Takiar
• Reporter: Joe McDonnell
• Created: 17/Apr/19 00:28
• Updated: 05/Oct/20 16:00
• Resolved: 14/May/19 14:36

Description

The file handle cache is currently disabled for S3, as the S3 connector needed to implement proper unbuffer support. Now that https://issues.apache.org/jira/browse/HADOOP-14747 is fixed, Impala should provide an option to cache S3 file handles.

This is particularly important for data caching, as accessing the data cache happens after obtaining a file handle. If getting a file handle is slow, the caching will be less effective.

Issue Links

• relates to: IMPALA-7265 Cache remote file handles
• relates to: IMPALA-10202 Enable file handle cache for ABFS files

Sub-Tasks

• Impala Doc: the file handle cache now supports S3: Closed