Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11132

Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2, 2.8.2
    • Component/s: fs/adl, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Azure data lake file system supports traversal access on file/folder and demands execute permission on parent for getFileStatus access. Ref HDFS-9552.

      testMkdirsFailsForSubdirectoryOfExistingFile contract test expectation fails with AcccessControlException when exists(...) check for sub-directory present under file.

      Expected : exists(...) to handle AcccessControlException and ignore during the check for sub-directory present under file.

      1. HDFS-11132.001.patch
        4 kB
        Vishwajeet Dusane
      2. HDFS-11132.branch-2.001.patch
        4 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Contract test testMkdirsFailsForSubdirectoryOfExistingFile under FileContextMainOperationsBaseTest and FileSystemContractBaseTest. Existing behavior of contract test is to create file with permission 644. and once the mkdirs is failed to create directory under the file, ensure using getFileStatus the directory does not exist under the file.

          On Azure data lake file system, ACL are default enabled on file/folder level as per guideline from HDFS-9552. and requires execute permission on parent to retrieve file information. Since the file does not have execute permission, getFileStatus fails with AccessControlException.

          Proposal is to handle AccessControlException and ignore as expected.

          FileContextMainOperationsBaseTest.java
          public void testMkdirsFailsForSubdirectoryOfExistingFile() throws Exception {
          ...
          Path testSubDir = path("/test/hadoop/file/subdir");
          ...
          
          // PROPOSED - IGNORE AS EXPECTED FOR FS WHEN AccessControlException is thrown 
              try {
                assertFalse(fs.exists(testSubDir));
              } catch (AccessControlException e) {
                //expected
              }
          
              Path testDeepSubDir = path("/test/hadoop/file/deep/sub/dir");
          
          ...
          
          // PROPOSED - IGNORE AS EXPECTED WHEN AccessControlException is thrown 
              try {
                assertFalse(fs.exists(testDeepSubDir));
              } catch (AccessControlException e) {
                // expected
              }
            }
          

          Steve Loughran, Chris Nauroth and Chris Douglas - What is your thought on this ?

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Contract test testMkdirsFailsForSubdirectoryOfExistingFile under FileContextMainOperationsBaseTest and FileSystemContractBaseTest . Existing behavior of contract test is to create file with permission 644. and once the mkdirs is failed to create directory under the file, ensure using getFileStatus the directory does not exist under the file. On Azure data lake file system, ACL are default enabled on file/folder level as per guideline from HDFS-9552 . and requires execute permission on parent to retrieve file information. Since the file does not have execute permission, getFileStatus fails with AccessControlException . Proposal is to handle AccessControlException and ignore as expected. FileContextMainOperationsBaseTest.java public void testMkdirsFailsForSubdirectoryOfExistingFile() throws Exception { ... Path testSubDir = path( "/test/hadoop/file/subdir" ); ... // PROPOSED - IGNORE AS EXPECTED FOR FS WHEN AccessControlException is thrown try { assertFalse(fs.exists(testSubDir)); } catch (AccessControlException e) { //expected } Path testDeepSubDir = path( "/test/hadoop/file/deep/sub/dir" ); ... // PROPOSED - IGNORE AS EXPECTED WHEN AccessControlException is thrown try { assertFalse(fs.exists(testDeepSubDir)); } catch (AccessControlException e) { // expected } } Steve Loughran , Chris Nauroth and Chris Douglas - What is your thought on this ?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I see: there's a file up the tree, and because that file doesn't have exec permission, you can't call getFileStatus further down.

          I agree with your proposal; just have the "expected" section have a comment that names this JIRA, something like

           // HDFS-1132 checks on paths under files may be rejected by file permission logic
          
          Show
          stevel@apache.org Steve Loughran added a comment - I see: there's a file up the tree, and because that file doesn't have exec permission, you can't call getFileStatus further down. I agree with your proposal; just have the "expected" section have a comment that names this JIRA, something like // HDFS-1132 checks on paths under files may be rejected by file permission logic
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks you Steve Loughran, attached patch.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks you Steve Loughran , attached patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 6m 49s trunk passed
          +1 compile 9m 33s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 18s trunk passed
          +1 findbugs 1m 24s trunk passed
          +1 javadoc 0m 46s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 13s the patch passed
          +1 javac 9m 13s the patch passed
          +1 checkstyle 0m 29s the patch passed
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 47s the patch passed
          +1 unit 8m 25s hadoop-common in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          44m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11132
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838663/HDFS-11132.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 08009b62f16e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 96f4392
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17543/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17543/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 49s trunk passed +1 compile 9m 33s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 24s trunk passed +1 javadoc 0m 46s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 13s the patch passed +1 javac 9m 13s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 47s the patch passed +1 unit 8m 25s hadoop-common in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 44m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11132 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838663/HDFS-11132.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 08009b62f16e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 96f4392 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17543/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17543/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          Committed to trunk branch. Thanks for your contribution, Vishwajeet Dusane.

          Show
          liuml07 Mingliang Liu added a comment - +1 Committed to trunk branch. Thanks for your contribution, Vishwajeet Dusane .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10925 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10925/)
          HDFS-11132. Allow AccessControlException in contract tests when (liuml07: rev 19f373a46b2abb7a575f7884a9c7443b8ed67cd3)

          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10925 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10925/ ) HDFS-11132 . Allow AccessControlException in contract tests when (liuml07: rev 19f373a46b2abb7a575f7884a9c7443b8ed67cd3) (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextMainOperationsBaseTest.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemContractBaseTest.java
          Hide
          vishwajeet.dusane Vishwajeet Dusane added a comment -

          Thanks a lot Mingliang Liu for quick turnaround and pushing this patch through.

          Show
          vishwajeet.dusane Vishwajeet Dusane added a comment - Thanks a lot Mingliang Liu for quick turnaround and pushing this patch through.
          Hide
          jzhuge John Zhuge added a comment -

          This JIRA belongs to Hadoop Common. Is it too late to change its Project?

          Show
          jzhuge John Zhuge added a comment - This JIRA belongs to Hadoop Common. Is it too late to change its Project?
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi John Zhuge, yes, this should belong to Hadoop Common. I don't know about the move though as the JIRA number will be different from the commit log. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Hi John Zhuge , yes, this should belong to Hadoop Common. I don't know about the move though as the JIRA number will be different from the commit log. Thanks,
          Hide
          jzhuge John Zhuge added a comment - - edited

          Re-open to attach HDFS-11132.branch-2.001.patch with trivial conflicts.

          Show
          jzhuge John Zhuge added a comment - - edited Re-open to attach HDFS-11132 .branch-2.001.patch with trivial conflicts.
          Hide
          jzhuge John Zhuge added a comment -

          Trigger pre-commit tests on branch-2

          Show
          jzhuge John Zhuge added a comment - Trigger pre-commit tests on branch-2
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 15m 56s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 8m 57s branch-2 passed
          +1 compile 7m 36s branch-2 passed with JDK v1.8.0_121
          +1 compile 8m 14s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 32s branch-2 passed
          +1 mvnsite 1m 8s branch-2 passed
          +1 mvneclipse 0m 18s branch-2 passed
          +1 findbugs 2m 1s branch-2 passed
          +1 javadoc 0m 57s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 1m 6s branch-2 passed with JDK v1.7.0_121
          +1 mvninstall 0m 45s the patch passed
          +1 compile 7m 28s the patch passed with JDK v1.8.0_121
          +1 javac 7m 28s the patch passed
          +1 compile 7m 51s the patch passed with JDK v1.7.0_121
          +1 javac 7m 51s the patch passed
          +1 checkstyle 0m 32s the patch passed
          +1 mvnsite 1m 11s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 7s the patch passed with JDK v1.7.0_121
          +1 unit 13m 27s hadoop-common in the patch passed with JDK v1.7.0_121.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          94m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11132
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859684/HDFS-11132.branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 30fa42fd7577 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 69f34e8
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18778/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18778/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 15m 56s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 8m 57s branch-2 passed +1 compile 7m 36s branch-2 passed with JDK v1.8.0_121 +1 compile 8m 14s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 32s branch-2 passed +1 mvnsite 1m 8s branch-2 passed +1 mvneclipse 0m 18s branch-2 passed +1 findbugs 2m 1s branch-2 passed +1 javadoc 0m 57s branch-2 passed with JDK v1.8.0_121 +1 javadoc 1m 6s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 45s the patch passed +1 compile 7m 28s the patch passed with JDK v1.8.0_121 +1 javac 7m 28s the patch passed +1 compile 7m 51s the patch passed with JDK v1.7.0_121 +1 javac 7m 51s the patch passed +1 checkstyle 0m 32s the patch passed +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 20s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_121 +1 unit 13m 27s hadoop-common in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 94m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11132 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859684/HDFS-11132.branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 30fa42fd7577 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 69f34e8 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18778/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18778/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

            People

            • Assignee:
              vishwajeet.dusane Vishwajeet Dusane
              Reporter:
              vishwajeet.dusane Vishwajeet Dusane
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development