[IMPALA-4611] Checking perms on S3 files is a very expensive no-op

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.9.0
• Component/s: Catalog
• Labels: performance s3
• Assignee: Unassigned
• Reporter: Sailesh Mukil
• Created: 07/Dec/16 02:11
• Updated: 26/Jan/17 21:02
• Resolved: 13/Dec/16 05:44

Description

After getting in IMPALA-4172/IMPALA-3653, we expected good gains in S3 metadata loading. However, as mmokhtar found recently, we spend a lot of time sending requests to S3 for every file to check the ACL permissions associated with it.

However, on S3, there are no Hadoop style ACLs but only object-level ACLs which are associated with the AWS credentials accessing the files (objects) and we cannot yet check or set permissions with the S3AFileSystem.

This means that we're wasting a lot of time waiting for S3 only to get back the same standard response every time. We can skip these checks when we're working with S3 files just the same way we skip 'inherit permissions' while creating new S3 files.

Issue Links

• relates to: IMPALA-3482 S3: Consider bulk listing of files in the catalog vs individually accessing them