Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9552

Document types of permission checks performed for HDFS operations.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: documentation
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The HDFS permissions guide discusses our use of a POSIX-like model with read, write and execute permissions associated with users, groups and the catch-all other class. However, there is no documentation that describes exactly what permission checks are performed by user-facing HDFS operations. This is a frequent source of questions, so it would be good to document this.

      1. hadoop-site.tar.bz2
        7.52 MB
        Chris Nauroth
      2. HDFS-9552.001.patch
        6 kB
        Chris Nauroth
      3. HDFS-9552.002.patch
        7 kB
        Chris Nauroth
      4. HDFS-9552.003.patch
        7 kB
        Chris Nauroth

        Activity

        Hide
        cnauroth Chris Nauroth added a comment -

        I'm attaching a patch that adds a table with the significant HDFS operations, and the access checks performed for those operations. I've also attached a tarball containing a full site build with the patch in case anyone prefers to review the HTML rendering.

        Show
        cnauroth Chris Nauroth added a comment - I'm attaching a patch that adds a table with the significant HDFS operations, and the access checks performed for those operations. I've also attached a tarball containing a full site build with the patch in case anyone prefers to review the HTML rendering.
        Hide
        cnauroth Chris Nauroth added a comment -

        I'm attaching patch v002 with a few additional clarifications about concat and setOwner.

        Show
        cnauroth Chris Nauroth added a comment - I'm attaching patch v002 with a few additional clarifications about concat and setOwner.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Nice work Chris Nauroth, thanks for documenting this.

        Should this be WRITE (target) instead of WRITE (source)?

        concat                | NO [2]    | WRITE (source) | N/A                 | READ (source), WRITE (destination) | N/A
        

        The rest looks good to me.

        Show
        arpitagarwal Arpit Agarwal added a comment - Nice work Chris Nauroth , thanks for documenting this. Should this be WRITE (target) instead of WRITE (source) ? concat | NO [2] | WRITE (source) | N/A | READ (source), WRITE (destination) | N/A The rest looks good to me.
        Hide
        cnauroth Chris Nauroth added a comment -

        Arpit Agarwal, thank you for the review.

        I think the concat entry as already written is accurate. FSDirConcatOp#verifySrcFiles contains this line:

                fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete
        

        I think the rationale is that since the original source files don't exist after the concat completes, it's like a delete of those inodes, so it ought to enforce write on the parent just like delete.

        Show
        cnauroth Chris Nauroth added a comment - Arpit Agarwal , thank you for the review. I think the concat entry as already written is accurate. FSDirConcatOp#verifySrcFiles contains this line: fsd.checkParentAccess(pc, iip, FsAction.WRITE); // for delete I think the rationale is that since the original source files don't exist after the concat completes, it's like a delete of those inodes, so it ought to enforce write on the parent just like delete.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks. That makes sense. +1 pending Jenkins.

        Nitpick you may consider fixing during commit, source should be sources since the check is enforced for each source file.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks. That makes sense. +1 pending Jenkins. Nitpick you may consider fixing during commit, source should be sources since the check is enforced for each source file.
        Hide
        cnauroth Chris Nauroth added a comment -

        Nitpick you may consider fixing during commit, source should be sources since the check is enforced for each source file.

        That's a good idea. Thanks! Here is patch v003 with that change.

        Show
        cnauroth Chris Nauroth added a comment - Nitpick you may consider fixing during commit, source should be sources since the check is enforced for each source file. That's a good idea. Thanks! Here is patch v003 with that change.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Thanks, +1 for the v003 patch pending Jenkins.

        Show
        arpitagarwal Arpit Agarwal added a comment - Thanks, +1 for the v003 patch pending Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 mvnsite 0m 53s trunk passed
        +1 mvnsite 0m 49s the patch passed
        -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        2m 14s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778130/HDFS-9552.003.patch
        JIRA Issue HDFS-9552
        Optional Tests asflicense mvnsite
        uname Linux 41534912c4b7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3c0adac
        whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13908/artifact/patchprocess/whitespace-eol.txt
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Max memory used 29MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13908/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 mvnsite 0m 53s trunk passed +1 mvnsite 0m 49s the patch passed -1 whitespace 0m 0s The patch has 58 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 2m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12778130/HDFS-9552.003.patch JIRA Issue HDFS-9552 Optional Tests asflicense mvnsite uname Linux 41534912c4b7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c0adac whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/13908/artifact/patchprocess/whitespace-eol.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 29MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13908/console This message was automatically generated.
        Hide
        cnauroth Chris Nauroth added a comment -

        The whitespace warning was triggered on a file not included in my patch. This is because the site build copies hdfs-default.xml to src/site/resources. That could be considered a build bug. It's bad form for the build to write into src.

        I'll commit this.

        Show
        cnauroth Chris Nauroth added a comment - The whitespace warning was triggered on a file not included in my patch. This is because the site build copies hdfs-default.xml to src/site/resources. That could be considered a build bug. It's bad form for the build to write into src. I'll commit this.
        Hide
        cnauroth Chris Nauroth added a comment -

        I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the review.

        Show
        cnauroth Chris Nauroth added a comment - I have committed this to trunk, branch-2 and branch-2.8. Arpit, thank you for the review.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8985/)
        HDFS-9552. Document types of permission checks performed for HDFS (cnauroth: rev 724e71f63b35695482642ed7aedacd70d08e6cb1)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsPermissionsGuide.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8985 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8985/ ) HDFS-9552 . Document types of permission checks performed for HDFS (cnauroth: rev 724e71f63b35695482642ed7aedacd70d08e6cb1) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsPermissionsGuide.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

          People

          • Assignee:
            cnauroth Chris Nauroth
            Reporter:
            cnauroth Chris Nauroth
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development