Details

    • Target Version/s:
    • Release Note:
      The HdfsAdmin erasure coding APIs (set, unset, get) are now usable by non-superusers based on appropriate file and directory permissions.

      Description

      We should allow to enforce authorization policy to protect administration operations for EC zone and schemas as such behaviors would impact too much for a system.

      1. hdfs-8112.001.patch
        11 kB
        Andrew Wang
      2. hdfs-8112.002.patch
        27 kB
        Andrew Wang
      3. hdfs-8112.003.patch
        27 kB
        Andrew Wang

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11342 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11342/)
          HDFS-8112. Relax permission checking for EC related operations. (wang: rev 3085a604300ed76d06a0011bd5555e419897b6cd)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11342 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11342/ ) HDFS-8112 . Relax permission checking for EC related operations. (wang: rev 3085a604300ed76d06a0011bd5555e419897b6cd) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirErasureCodingOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Committed based on Rakesh's +1, thanks for reviewing Kai and Rakesh!

          Show
          andrew.wang Andrew Wang added a comment - Committed based on Rakesh's +1, thanks for reviewing Kai and Rakesh!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 27s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 43s trunk passed
          +1 javadoc 0m 40s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 354 unchanged - 7 fixed = 355 total (was 361)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 6s the patch passed
          +1 javadoc 0m 42s the patch passed
          -1 unit 75m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          100m 22s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855937/hdfs-8112.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5e2b8b7a4e37 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 490abfb
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18544/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18544/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18544/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18544/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 27s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 43s trunk passed +1 javadoc 0m 40s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 37s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 354 unchanged - 7 fixed = 355 total (was 361) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 6s the patch passed +1 javadoc 0m 42s the patch passed -1 unit 75m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 100m 22s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855937/hdfs-8112.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5e2b8b7a4e37 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 490abfb Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18544/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18544/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18544/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18544/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the reviews! Fixed the checkstyles and renamed the test to not say "Zone".

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the reviews! Fixed the checkstyles and renamed the test to not say "Zone".
          Hide
          rakeshr Rakesh R added a comment -

          Thank you very much Andrew Wang for posting the patch. Apart from Kai's comment, changes looks good to me. +1

          Show
          rakeshr Rakesh R added a comment - Thank you very much Andrew Wang for posting the patch. Apart from Kai's comment, changes looks good to me. +1
          Hide
          drankye Kai Zheng added a comment -

          Thanks Andrew Wang for the quick taking on this. The patch looks pretty good. A little minor towards +1, in addition to the check styles, could you refine testCreationErasureCodingZoneWithInvalidPolicy by the way, since we don't favor ErasureCodingZone concept any more.

          Show
          drankye Kai Zheng added a comment - Thanks Andrew Wang for the quick taking on this. The patch looks pretty good. A little minor towards +1, in addition to the check styles, could you refine testCreationErasureCodingZoneWithInvalidPolicy by the way, since we don't favor ErasureCodingZone concept any more.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 1s trunk passed
          +1 compile 0m 56s trunk passed
          +1 checkstyle 0m 45s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 0m 46s trunk passed
          +1 mvninstall 0m 59s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 354 unchanged - 7 fixed = 359 total (was 361)
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 93m 58s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          121m 47s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855718/hdfs-8112.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f62e9a143d25 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a97833e
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18523/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18523/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 1s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 45s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 59s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed -0 checkstyle 0m 36s hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 354 unchanged - 7 fixed = 359 total (was 361) +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 93m 58s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 121m 47s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855718/hdfs-8112.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f62e9a143d25 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a97833e Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18523/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18523/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18523/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 35s trunk passed
          +1 compile 0m 57s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 0m 44s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 45s the patch passed
          +1 javac 0m 45s the patch passed
          -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 197 unchanged - 0 fixed = 201 total (was 197)
          +1 mvnsite 0m 50s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 53s the patch passed
          +1 javadoc 0m 37s the patch passed
          -1 unit 73m 27s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          100m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-8112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855700/hdfs-8112.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 23e5f0296da0 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a97833e
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18522/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18522/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 35s trunk passed +1 compile 0m 57s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 35s hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 197 unchanged - 0 fixed = 201 total (was 197) +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 53s the patch passed +1 javadoc 0m 37s the patch passed -1 unit 73m 27s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 100m 11s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-8112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12855700/hdfs-8112.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 23e5f0296da0 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a97833e Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18522/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18522/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18522/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          One more rev attached, changes:

          • Realized I missed unsetECPolicy, handled now with write perms
          • Did some refactors/code cleanups in FSDirErasureCodingOp: made some things private, renamed the unprotected getECPolicy method, removed unused hasECPolicy method, inlined getINodesInPath method only used once for permission clarity
          • Added testing for all EC related methods exposed in HdfsAdmin
          Show
          andrew.wang Andrew Wang added a comment - One more rev attached, changes: Realized I missed unsetECPolicy, handled now with write perms Did some refactors/code cleanups in FSDirErasureCodingOp: made some things private, renamed the unprotected getECPolicy method, removed unused hasECPolicy method, inlined getINodesInPath method only used once for permission clarity Added testing for all EC related methods exposed in HdfsAdmin
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Rakesh, hope you don't mind but I saw this on the must-do list and thought I'd make a patch, attached.

          Show
          andrew.wang Andrew Wang added a comment - Hi Rakesh, hope you don't mind but I saw this on the must-do list and thought I'd make a patch, attached.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hello Rakesh R are you still working on this? This is a blocker issue for Hadoop 3 so want to ping you to see if you need help here.
          We should also change the summary of this jira as the scope has changed due to EC evolution.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hello Rakesh R are you still working on this? This is a blocker issue for Hadoop 3 so want to ping you to see if you need help here. We should also change the summary of this jira as the scope has changed due to EC evolution.
          Hide
          drankye Kai Zheng added a comment -

          Changing EC policies of a directory should be equivalent to updating its xattr.

          Right, or equivalent to setStoragePolicy of a directory.

          Show
          drankye Kai Zheng added a comment - Changing EC policies of a directory should be equivalent to updating its xattr. Right, or equivalent to setStoragePolicy of a directory.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          There are a few EC related APIs: setErasureCodingPolicy, unsetErasureCodingPolicy, getErasureCodingPolicy and getErasureCodingPolicies. HDFS-7859 will add addErasureCodingPolicy.

          Except for getErasureCodingPolicy and getErasureCodingPolicies, all EC APIs requires super user permission. This is not right. Changing EC policies of a directory should be equivalent to updating its xattr.

          Show
          jojochuang Wei-Chiu Chuang added a comment - There are a few EC related APIs: setErasureCodingPolicy, unsetErasureCodingPolicy, getErasureCodingPolicy and getErasureCodingPolicies. HDFS-7859 will add addErasureCodingPolicy. Except for getErasureCodingPolicy and getErasureCodingPolicies, all EC APIs requires super user permission. This is not right. Changing EC policies of a directory should be equivalent to updating its xattr.
          Hide
          andrew.wang Andrew Wang added a comment -

          Marking ec must do's as blockers for alpha3

          Show
          andrew.wang Andrew Wang added a comment - Marking ec must do's as blockers for alpha3
          Hide
          andrew.wang Andrew Wang added a comment -

          Hi folks, do we want to revive this issue? Now it's policies rather than zones, but the restriction for setting EC policies is still superuser-only.

          I'd like to see us open this up to normal users too if possible.

          Show
          andrew.wang Andrew Wang added a comment - Hi folks, do we want to revive this issue? Now it's policies rather than zones, but the restriction for setting EC policies is still superuser-only. I'd like to see us open this up to normal users too if possible.
          Hide
          zhangyongxyz Yong Zhang added a comment -

          Thank you Rakesh R to clear the background about this jira.
          These days I work on append feature and want a ec file to test the append, and find creating ec zone need superuser privilege which does not meet user scenario, so I submit HDFS-8333, but then find you have worked on it, so want to discuss with you.

          Show
          zhangyongxyz Yong Zhang added a comment - Thank you Rakesh R to clear the background about this jira. These days I work on append feature and want a ec file to test the append, and find creating ec zone need superuser privilege which does not meet user scenario, so I submit HDFS-8333 , but then find you have worked on it, so want to discuss with you.
          Hide
          rakeshr Rakesh R added a comment -

          Thank you Yong Zhang for the comments and bringing up the use case.

          IIUC you are saying ErasureCoding APIs can check user permission against the ACLs of the FSDirectory. Also, we can define the File system actions(r, w, etc) as per EC operations. When raising this jira Kai Zheng's idea is to enforce protection policy at the protocol layer Hadoop Service Level Authorization which ensures only privileged users/admins to be able to perform the operations. Initially we thought all DFS commands for EC should be in client protocol for this discussion. But on a second thought, there may come new APIs in other protocol as well. So we have decided to take up this jira later(could leave for other issues or discussions) and is the reason I didn't give much focus on this jira. I could see today you have raised HDFS-8333 to discuss the Create EC zone API user privileges. Probably we could listen the thoughts from others and take up this task accordingly.

          Show
          rakeshr Rakesh R added a comment - Thank you Yong Zhang for the comments and bringing up the use case. IIUC you are saying ErasureCoding APIs can check user permission against the ACLs of the FSDirectory. Also, we can define the File system actions(r, w, etc) as per EC operations. When raising this jira Kai Zheng 's idea is to enforce protection policy at the protocol layer Hadoop Service Level Authorization which ensures only privileged users/admins to be able to perform the operations. Initially we thought all DFS commands for EC should be in client protocol for this discussion. But on a second thought, there may come new APIs in other protocol as well. So we have decided to take up this jira later(could leave for other issues or discussions) and is the reason I didn't give much focus on this jira. I could see today you have raised HDFS-8333 to discuss the Create EC zone API user privileges. Probably we could listen the thoughts from others and take up this task accordingly.
          Hide
          zhangyongxyz Yong Zhang added a comment -

          Hi Rakesh R, are you still working on it?
          I suggest that we can check if user has permission to createErasureCodingZone via checkPathAccess with FsAction.ALL, and getErasureCodingInfo and getErasureCodingZoneInfo via checkPathAccess with FsAction.READ.
          all there 3 API are common operation in multiple tenant scenario.

          Show
          zhangyongxyz Yong Zhang added a comment - Hi Rakesh R , are you still working on it? I suggest that we can check if user has permission to createErasureCodingZone via checkPathAccess with FsAction.ALL, and getErasureCodingInfo and getErasureCodingZoneInfo via checkPathAccess with FsAction.READ. all there 3 API are common operation in multiple tenant scenario.
          Hide
          rakeshr Rakesh R added a comment -

          As per the discussion with Kai Zheng, the idea of this task is to revisit all the EC command/API operations and refine this aspect once the whole feature is solid. There could be cases where some operations may be good available for non-super admin users.

          Show
          rakeshr Rakesh R added a comment - As per the discussion with Kai Zheng , the idea of this task is to revisit all the EC command/API operations and refine this aspect once the whole feature is solid. There could be cases where some operations may be good available for non-super admin users.
          Hide
          rakeshr Rakesh R added a comment -

          I've gone through the code, presently the following operations related to EC in the FSNameSystem:

          • createErasureCodingZone
          • getErasureCodingInfo
          • getErasureCodingZoneInfo

          Out of these, createErasureCodingZone already has checkSuperuserPrivilege performed.

          Show
          rakeshr Rakesh R added a comment - I've gone through the code, presently the following operations related to EC in the FSNameSystem: createErasureCodingZone getErasureCodingInfo getErasureCodingZoneInfo Out of these, createErasureCodingZone already has checkSuperuserPrivilege performed.
          Hide
          drankye Kai Zheng added a comment -

          I'm wondering if the superuser privilege is too restricted or not to operations for both EC zone and schemas. I thought it's good for now to have. Thanks!

          Show
          drankye Kai Zheng added a comment - I'm wondering if the superuser privilege is too restricted or not to operations for both EC zone and schemas. I thought it's good for now to have. Thanks!
          Hide
          rakeshr Rakesh R added a comment -

          We should simply call checkSuperuserPrivilege in FSNamesystem for the admin operations.

          Thanks Tsz Wo Nicholas Sze for the hint.

          Yes, am working and will post a patch soon.

          Show
          rakeshr Rakesh R added a comment - We should simply call checkSuperuserPrivilege in FSNamesystem for the admin operations. Thanks Tsz Wo Nicholas Sze for the hint. Yes, am working and will post a patch soon.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          BTW, are you still working on this?

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - BTW, are you still working on this?
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          We should simply call checkSuperuserPrivilege in FSNamesystem for the admin operations.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - We should simply call checkSuperuserPrivilege in FSNamesystem for the admin operations.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks again for the details. I will go through it. Kindly assign the issue to me.

          Show
          rakeshr Rakesh R added a comment - Thanks again for the details. I will go through it. Kindly assign the issue to me.
          Hide
          drankye Kai Zheng added a comment -

          Hmm, maybe you could learn a little bit about Hadoop Service Level Authorization I guess? In codes, please see RefreshAuthorizationPolicyProtocol, which ensures only privileged users/admins to be able to update and load hadoop-policy.xml ACL file. This is nothing special from that. Hope this helps, thanks.

          Show
          drankye Kai Zheng added a comment - Hmm, maybe you could learn a little bit about Hadoop Service Level Authorization I guess? In codes, please see RefreshAuthorizationPolicyProtocol , which ensures only privileged users/admins to be able to update and load hadoop-policy.xml ACL file. This is nothing special from that. Hope this helps, thanks.
          Hide
          rakeshr Rakesh R added a comment -

          Thanks a lot Kai Zheng. Yes, I'm happy to take up this. BTW I'd like to know any draft idea/thought that comes up in your mind - about the possible way to enforce the authorization policy in order to protect EC zone and schemas.

          Show
          rakeshr Rakesh R added a comment - Thanks a lot Kai Zheng . Yes, I'm happy to take up this. BTW I'd like to know any draft idea/thought that comes up in your mind - about the possible way to enforce the authorization policy in order to protect EC zone and schemas.
          Hide
          drankye Kai Zheng added a comment -

          Hi Rakesh R,

          Enforce protection policy for managing EC schemas is an important aspect of HDFS-7866. If you're interested in it, please feel free to take it. Thanks.

          Show
          drankye Kai Zheng added a comment - Hi Rakesh R , Enforce protection policy for managing EC schemas is an important aspect of HDFS-7866 . If you're interested in it, please feel free to take it. Thanks.

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              drankye Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development