Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4611

Checking perms on S3 files is a very expensive no-op

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.7.0
    • Impala 2.9.0
    • Catalog

    Description

      After getting in IMPALA-4172/IMPALA-3653, we expected good gains in S3 metadata loading. However, as mmokhtar found recently, we spend a lot of time sending requests to S3 for every file to check the ACL permissions associated with it.

      However, on S3, there are no Hadoop style ACLs but only object-level ACLs which are associated with the AWS credentials accessing the files (objects) and we cannot yet check or set permissions with the S3AFileSystem.

      This means that we're wasting a lot of time waiting for S3 only to get back the same standard response every time. We can skip these checks when we're working with S3 files just the same way we skip 'inherit permissions' while creating new S3 files.

      Attachments

        1. invalidate_cs_3.jfr
          1.90 MB
          Sailesh Mukil

        Issue Links

          Activity

            People

              Unassigned Unassigned
              sailesh Sailesh Mukil
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: