Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4611

Checking perms on S3 files is a very expensive no-op

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.7.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Catalog
    • Labels:

      Description

      After getting in IMPALA-4172/IMPALA-3653, we expected good gains in S3 metadata loading. However, as Mostafa Mokhtar found recently, we spend a lot of time sending requests to S3 for every file to check the ACL permissions associated with it.

      However, on S3, there are no Hadoop style ACLs but only object-level ACLs which are associated with the AWS credentials accessing the files (objects) and we cannot yet check or set permissions with the S3AFileSystem.

      This means that we're wasting a lot of time waiting for S3 only to get back the same standard response every time. We can skip these checks when we're working with S3 files just the same way we skip 'inherit permissions' while creating new S3 files.

        Attachments

        1. invalidate_cs_3.jfr
          1.90 MB
          Sailesh Mukil

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sailesh Sailesh Mukil
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: