Details

      Description

      For the ranger-wasb integration, we need owner information from the metadata information of the files/folders to be passed along to the ranger authorizer.

      This patch contains the changes related to retrieving the owner from metadata and making it available for ranger plugin that is integrated with wasb.

      1. HADOOP-14442.patch
        33 kB
        Varada Hemeswari
      2. HADOOP-14442.1.patch
        34 kB
        Varada Hemeswari

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11792 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11792/)
        HADOOP-14442. Owner support for ranger-wasb integration. Contributed by (liuml07: rev 89bb8bfe582ba85566cede321b233bb642f1c675)

        • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java
        • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
        • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorizationWithOwner.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11792 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11792/ ) HADOOP-14442 . Owner support for ranger-wasb integration. Contributed by (liuml07: rev 89bb8bfe582ba85566cede321b233bb642f1c675) (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorizationWithOwner.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 on the v1 patch.

        Can you update the description for root directory case? Thanks Varada Hemeswari.

        Show
        liuml07 Mingliang Liu added a comment - +1 on the v1 patch. Can you update the description for root directory case? Thanks Varada Hemeswari .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 14m 4s trunk passed
        +1 compile 0m 20s trunk passed
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 19s trunk passed
        +1 mvneclipse 0m 24s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 14s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 15s the patch passed
        +1 javac 0m 15s the patch passed
        +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)
        +1 mvnsite 0m 16s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 0m 33s the patch passed
        +1 javadoc 0m 12s the patch passed
        +1 unit 1m 41s hadoop-azure in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        21m 31s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14442
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870075/HADOOP-14442.1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 5aa14174b584 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aea4293
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 14m 4s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 19s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 15s the patch passed +1 javac 0m 15s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34) +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 12s the patch passed -1 whitespace 0m 0s The patch has 4 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 33s the patch passed +1 javadoc 0m 12s the patch passed +1 unit 1m 41s hadoop-azure in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 21m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14442 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870075/HADOOP-14442.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5aa14174b584 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aea4293 Default Java 1.8.0_131 findbugs v3.1.0-RC1 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12400/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        Can you briefly describe the owner mode here in the "Description" section?

        1. As we're using the slf4j, we don't need the String.format for parameters. Instead, we can use placeholder. E.g.
          - LOG.debug(String.format("Cannot find file/folder - '%s'. Returning owner as empty string", absolutePath));
          +LOG.debug("Cannot find file/folder - '{}'. Returning owner as empty string", absolutePath);
          
        2. The ex to be thrown include the error message for easier debugging. The error message can be the same as the LOG.error() before it.
          getOwnerForPath()
          3200	    } catch(IOException ex) {
          3201	
          3202	          Throwable innerException = NativeAzureFileSystemHelper.checkForAzureStorageException(ex);
          3203	          boolean isfileNotFoundException = innerException instanceof StorageException
          3204	            && NativeAzureFileSystemHelper.isFileNotFoundException((StorageException) innerException);
          3205	
          3206	          // should not throw when the exception is related to blob/container/file/folder not found
          3207	          if (!isfileNotFoundException) {
          3208	            LOG.error(String.format("Could not retrieve owner information for path - '%s'", absolutePath));
          3209	            throw ex;
          3210	          }
          3211	      }
          3212	    return owner;
          
        3. It's better to use UserGroupInformation.createUserForTesting.doAs() clause in testOwnerPermissionNegative instead of simply initialize the current user name. Is it possible to do that?
        4. Nit: the test path can use the test method (test case) name as the directory name, to avoid future conflicts. e.g. Path parentDir = new Path("/testOwnerPositive"); => final Path parentDir = new Path("testOwnerPermissionPositive");
        5. Nit: the checkstyle warnings seem related.
        Show
        liuml07 Mingliang Liu added a comment - Can you briefly describe the owner mode here in the "Description" section? As we're using the slf4j, we don't need the String.format for parameters. Instead, we can use placeholder. E.g. - LOG.debug( String .format( "Cannot find file/folder - '%s'. Returning owner as empty string" , absolutePath)); +LOG.debug( "Cannot find file/folder - '{}'. Returning owner as empty string" , absolutePath); The ex to be thrown include the error message for easier debugging. The error message can be the same as the LOG.error() before it. getOwnerForPath() 3200 } catch (IOException ex) { 3201 3202 Throwable innerException = NativeAzureFileSystemHelper.checkForAzureStorageException(ex); 3203 boolean isfileNotFoundException = innerException instanceof StorageException 3204 && NativeAzureFileSystemHelper.isFileNotFoundException((StorageException) innerException); 3205 3206 // should not throw when the exception is related to blob/container/file/folder not found 3207 if (!isfileNotFoundException) { 3208 LOG.error( String .format( "Could not retrieve owner information for path - '%s'" , absolutePath)); 3209 throw ex; 3210 } 3211 } 3212 return owner; It's better to use UserGroupInformation.createUserForTesting.doAs() clause in testOwnerPermissionNegative instead of simply initialize the current user name. Is it possible to do that? Nit: the test path can use the test method (test case) name as the directory name, to avoid future conflicts. e.g. Path parentDir = new Path("/testOwnerPositive"); => final Path parentDir = new Path("testOwnerPermissionPositive"); Nit: the checkstyle warnings seem related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 15m 41s trunk passed
        +1 compile 0m 19s trunk passed
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 21s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 0m 32s trunk passed
        +1 javadoc 0m 14s trunk passed
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 9 new + 34 unchanged - 0 fixed = 43 total (was 34)
        +1 mvnsite 0m 18s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 0s The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 0m 34s the patch passed
        +1 javadoc 0m 11s the patch passed
        +1 unit 1m 20s hadoop-azure in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        23m 12s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14442
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869875/HADOOP-14442.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f65a685322f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2e41f88
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
        whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/artifact/patchprocess/whitespace-eol.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 15m 41s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 9 new + 34 unchanged - 0 fixed = 43 total (was 34) +1 mvnsite 0m 18s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 14 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 0m 34s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 1m 20s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 23m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14442 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869875/HADOOP-14442.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f65a685322f0 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e41f88 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12396/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        vahemesw Varada Hemeswari added a comment -

        Mingliang Liu, Can you please review the patch attached.

        Show
        vahemesw Varada Hemeswari added a comment - Mingliang Liu , Can you please review the patch attached.

          People

          • Assignee:
            vahemesw Varada Hemeswari
            Reporter:
            vahemesw Varada Hemeswari
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Resolved:

              Development