Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      S3A does not match the implementation of HDFS in its handling of file ownership and permissions. Fundamental S3 limitations prevent it. This is a frequent source of confusion for end users. This issue proposes to document these known limitations.

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          Limitations as of right now:

          1. There is no notion of separate identity for different users. Once a user is authenticated to the S3 bucket, they effectively have full access to the file system tree.
          2. The file owner is always reported as the user running the current process.
          3. The group is always null (no group associated).
          4. Permissions are always 777 for directories and 666 for files.
          5. S3A performs no actual checks on the permissions.  It's just authentication to the entire S3 bucket, based on AWS credentials. It's possible that object ACLs are applied, but this is something enforced entirely within S3, not a feature of S3A.
          Show
          cnauroth Chris Nauroth added a comment - Limitations as of right now: There is no notion of separate identity for different users. Once a user is authenticated to the S3 bucket, they effectively have full access to the file system tree. The file owner is always reported as the user running the current process. The group is always null (no group associated). Permissions are always 777 for directories and 666 for files. S3A performs no actual checks on the permissions.  It's just authentication to the entire S3 bucket, based on AWS credentials. It's possible that object ACLs are applied, but this is something enforced entirely within S3, not a feature of S3A.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          In HADOOP-12774 I proposed making the owner UGI.getShortName(), that way in code running as a different user, you'd pick up the relevant user.

          not sure it matters though

          Show
          stevel@apache.org Steve Loughran added a comment - In HADOOP-12774 I proposed making the owner UGI.getShortName(), that way in code running as a different user, you'd pick up the relevant user. not sure it matters though
          Hide
          cnauroth Chris Nauroth added a comment -

          In HADOOP-12774 I proposed making the owner UGI.getShortName(), that way in code running as a different user, you'd pick up the relevant user.

          This is the same implementation choice that Azure Data Lake used recently in HADOOP-12666. Maybe it makes sense for the file system spec to state that file systems without support for ownership must fall back to UGI? Then, the limitation would at least be the same limitation across alternative file systems.

          Show
          cnauroth Chris Nauroth added a comment - In HADOOP-12774 I proposed making the owner UGI.getShortName(), that way in code running as a different user, you'd pick up the relevant user. This is the same implementation choice that Azure Data Lake used recently in HADOOP-12666 . Maybe it makes sense for the file system spec to state that file systems without support for ownership must fall back to UGI? Then, the limitation would at least be the same limitation across alternative file systems.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1 for a documented convention. UGI shortname for user & group in the absence of anything else.

          For blobstores, maybe one feature that could be queried is something about what permissions models are available, Unix-y user+group+other

          Show
          stevel@apache.org Steve Loughran added a comment - +1 for a documented convention. UGI shortname for user & group in the absence of anything else. For blobstores, maybe one feature that could be queried is something about what permissions models are available, Unix-y user+group+other
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user cnauroth opened a pull request:

          https://github.com/apache/hadoop/pull/138

          HADOOP-13309: Document S3A known limitations in file ownership and pe…

          …rmission model.

          Summary:

          • Update file system specification to describe that object stores may have a different authorization model than HDFS and traditional file systems.
          • Update hadoop-aws documentation to warn that S3A will return stub information for metadata related to ownership and permissions. I wrote this information from the assumption that the HADOOP-12774 change gets finished, so that one will have to get committed first.
          • Also update a few cosmetic things near the part of the hadoop-aws document that I changed.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cnauroth/hadoop-1 HADOOP-13309

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/138.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #138


          commit 5c2a05463523e4b101eea08611e036315c1bd63a
          Author: Chris Nauroth <cnauroth@apache.org>
          Date: 2016-10-10T22:51:21Z

          HADOOP-13309: Document S3A known limitations in file ownership and permission model.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user cnauroth opened a pull request: https://github.com/apache/hadoop/pull/138 HADOOP-13309 : Document S3A known limitations in file ownership and pe… …rmission model. Summary: Update file system specification to describe that object stores may have a different authorization model than HDFS and traditional file systems. Update hadoop-aws documentation to warn that S3A will return stub information for metadata related to ownership and permissions. I wrote this information from the assumption that the HADOOP-12774 change gets finished, so that one will have to get committed first. Also update a few cosmetic things near the part of the hadoop-aws document that I changed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cnauroth/hadoop-1 HADOOP-13309 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/138.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #138 commit 5c2a05463523e4b101eea08611e036315c1bd63a Author: Chris Nauroth <cnauroth@apache.org> Date: 2016-10-10T22:51:21Z HADOOP-13309 : Document S3A known limitations in file ownership and permission model.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          reviewed text, LGTM, +1

          Yetus hasn't looked at this —but being an .md only patch, there's not much for it to review

          Show
          stevel@apache.org Steve Loughran added a comment - reviewed text, LGTM, +1 Yetus hasn't looked at this —but being an .md only patch, there's not much for it to review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          0 mvndep 0m 55s Maven dependency ordering for branch
          +1 mvninstall 6m 33s branch-2 passed
          +1 mvnsite 1m 22s branch-2 passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvnsite 1m 18s the patch passed
          -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          11m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13309
          GITHUB PR https://github.com/apache/hadoop/pull/138
          Optional Tests asflicense mvnsite
          uname Linux 5f50f5a28fa8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / e341e51
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/artifact/patchprocess/whitespace-eol.txt
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. 0 mvndep 0m 55s Maven dependency ordering for branch +1 mvninstall 6m 33s branch-2 passed +1 mvnsite 1m 22s branch-2 passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvnsite 1m 18s the patch passed -1 whitespace 0m 0s The patch has 47 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 asflicense 0m 17s The patch does not generate ASF License warnings. 11m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13309 GITHUB PR https://github.com/apache/hadoop/pull/138 Optional Tests asflicense mvnsite uname Linux 5f50f5a28fa8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / e341e51 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/artifact/patchprocess/whitespace-eol.txt modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10745/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          That was a pre-commit run that I manually submitted at builds.apache.org. The whitespace warnings are not relevant.

          I'm going to wait until HADOOP-12774 gets committed first before I commit this one.

          Show
          cnauroth Chris Nauroth added a comment - That was a pre-commit run that I manually submitted at builds.apache.org. The whitespace warnings are not relevant. I'm going to wait until HADOOP-12774 gets committed first before I commit this one.
          Hide
          cnauroth Chris Nauroth added a comment -

          I submitted a fresh pre-commit run just to be sure, since a few patches have been committed ahead of this.

          https://builds.apache.org/job/PreCommit-HADOOP-Build/10888/

          Show
          cnauroth Chris Nauroth added a comment - I submitted a fresh pre-commit run just to be sure, since a few patches have been committed ahead of this. https://builds.apache.org/job/PreCommit-HADOOP-Build/10888/
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          0 mvndep 0m 51s Maven dependency ordering for branch
          +1 mvninstall 6m 33s branch-2 passed
          +1 mvnsite 1m 23s branch-2 passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvnsite 1m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          11m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HADOOP-13309
          GITHUB PR https://github.com/apache/hadoop/pull/138
          Optional Tests asflicense mvnsite
          uname Linux d462d6b59a8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 5c2f67b
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10888/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. 0 mvndep 0m 51s Maven dependency ordering for branch +1 mvninstall 6m 33s branch-2 passed +1 mvnsite 1m 23s branch-2 passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvnsite 1m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 11m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HADOOP-13309 GITHUB PR https://github.com/apache/hadoop/pull/138 Optional Tests asflicense mvnsite uname Linux d462d6b59a8d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 5c2f67b modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10888/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          cnauroth Chris Nauroth added a comment -

          I have committed this to trunk, branch-2 and branch-2.8. Steve, thank you for the code review.

          Show
          cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Steve, thank you for the code review.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10672/)
          HADOOP-13309. Document S3A known limitations in file ownership and (cnauroth: rev 309a43925c078ff51cdb6bd1273e6f91f43311cb)

          • (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10672 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10672/ ) HADOOP-13309 . Document S3A known limitations in file ownership and (cnauroth: rev 309a43925c078ff51cdb6bd1273e6f91f43311cb) (edit) hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md (edit) hadoop-common-project/hadoop-common/src/site/markdown/filesystem/introduction.md
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cnauroth closed the pull request at:

          https://github.com/apache/hadoop/pull/138

          Show
          githubbot ASF GitHub Bot added a comment - Github user cnauroth closed the pull request at: https://github.com/apache/hadoop/pull/138

            People

            • Assignee:
              cnauroth Chris Nauroth
              Reporter:
              cnauroth Chris Nauroth
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development