Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-15314

Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      In version 3.0.0, the checkPath method for dealing with paths prevents us from using the s3:// scheme when utilizing S3Guard. However, in our core-site.xml we have included 

        <property>
          <name>fs.s3.impl</name>
          <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
        </property>

      which should enforce that s3 prefixed paths go through s3a and are properly compatible with s3guard. We removed the assertion that paths use the s3a scheme (some of our paths use the s3 scheme) and our testing thus far with S3Guard enabled have been positive. We believe the assertion in checkPath is unnecessary and could be expanded to include the s3 and s3n schemes if not dropped altogether or altered in some other way. We're happy to develop and test a patch if the community is amenable to the change.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              djhoffman DJ Hoffman
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: