Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs/azure
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      As highlighted in HADOOP-13863, current implementation of WASB does not support authorization to any File System operations. This jira is created to add authorization support for WASB. The current approach is to enforce authorization via an external REST service (One approach could be to use component like Ranger to enforce authorization). The support for authorization would be hiding behind a configuration flag : "fs.azure.enable.authorization" and the remote service is expected to be provided via config : "fs.azure.remote.auth.service.url".

      The remote service is expected to provide support for the following REST call:

      {URL}/CHECK_AUTHORIZATION```

      An example request:{URL}

      /CHECK_AUTHORIZATION?wasb_absolute_path=<absolute_path>&operation_type=<operation type>&delegation_token=<delegation token>

      1. HADOOP-13930.001.patch
        44 kB
        Dushyanth
      2. HADOOP-13930.002.patch
        37 kB
        Dushyanth
      3. HADOOP-13930.003.patch
        139 kB
        Sivaguru Sankaridurg
      4. HADOOP-13930.004.patch
        139 kB
        Sivaguru Sankaridurg
      5. HADOOP-13930.005.patch
        68 kB
        Sivaguru Sankaridurg
      6. HADOOP-13930.006.patch
        68 kB
        Sivaguru Sankaridurg
      7. HADOOP-13930.007.patch
        68 kB
        Sivaguru Sankaridurg
      8. HADOOP-13930.008.patch
        69 kB
        Sivaguru Sankaridurg
      9. HADOOP-13930.009.patch
        68 kB
        Sivaguru Sankaridurg

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11360 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11360/)
          Revert "HADOOP-13930. Azure: Add Authorization support to WASB. (liuml07: rev 52d7d5aa1a303cf70519a61487641211f4267c6f)

          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (delete) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
          • (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/package.html
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java
          • (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java
          • (delete) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenIdentifier.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java
            HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by (liuml07: rev 686823529be09bea2a6cecb3503ef722017475bc)
          • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java
          • (edit) hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java
          • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
          • (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11360 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11360/ ) Revert " HADOOP-13930 . Azure: Add Authorization support to WASB. (liuml07: rev 52d7d5aa1a303cf70519a61487641211f4267c6f) (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (delete) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/package.html (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java (delete) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java (delete) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenIdentifier.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java HADOOP-13930 . Azure: Add Authorization support to WASB. Contributed by (liuml07: rev 686823529be09bea2a6cecb3503ef722017475bc) (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/test/resources/azure-test.xml (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          Thanks Mingliang.

          -Siva

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - Thanks Mingliang. -Siva
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11346 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11346/)
          HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by (liuml07: rev 6b7cd62b8cf12616b13142f2eb2cfc2f25796f0f)

          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java
          • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java
          • (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • (add) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenIdentifier.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java
          • (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • (add) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/package.html
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java
          • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11346 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11346/ ) HADOOP-13930 . Azure: Add Authorization support to WASB. Contributed by (liuml07: rev 6b7cd62b8cf12616b13142f2eb2cfc2f25796f0f) (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationException.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureStorageInterfaceImpl.java (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/MockWasbAuthorizerImpl.java (edit) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml (add) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizationOperations.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenIdentifier.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md (add) hadoop-tools/hadoop-azure/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/package.html (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbAuthorizerInterface.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
          Hide
          liuml07 Mingliang Liu added a comment -

          +1

          I have committed to trunk and branch-2 branches. Thanks for your contribution Sivaguru Sankaridurg and Dushyanth; thanks for review Steve.

          Show
          liuml07 Mingliang Liu added a comment - +1 I have committed to trunk and branch-2 branches. Thanks for your contribution Sivaguru Sankaridurg and Dushyanth ; thanks for review Steve.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - - edited

          Hi Steve Loughran, Mingliang Liu.

          Could you please review my latest patch ?
          For some reason tests in hadoop-common fail constantly, despite the fact that I have not made any changes in hadoop-common that are related to these test failures.

          These three tests keep failing constantly:
          https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/?auto_refresh=false

          I am seeing other build-outputs showing the same error - https://builds.apache.org/job/PreCommit-HADOOP-Build/11659/.
          I suppose it is not specific to my change. Could you please take a look and confirm ?

          Thanks
          Siva

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - - edited Hi Steve Loughran , Mingliang Liu . Could you please review my latest patch ? For some reason tests in hadoop-common fail constantly, despite the fact that I have not made any changes in hadoop-common that are related to these test failures. These three tests keep failing constantly: https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/?auto_refresh=false I am seeing other build-outputs showing the same error - https://builds.apache.org/job/PreCommit-HADOOP-Build/11659/ . I suppose it is not specific to my change. Could you please take a look and confirm ? Thanks Siva
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 13m 37s trunk passed
          +1 compile 14m 3s trunk passed
          +1 checkstyle 1m 56s trunk passed
          +1 mvnsite 1m 25s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 0s trunk passed
          +1 javadoc 1m 10s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 57s the patch passed
          +1 compile 11m 30s the patch passed
          +1 javac 11m 30s the patch passed
          +1 checkstyle 1m 57s the patch passed
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 2m 20s the patch passed
          +1 javadoc 0m 53s hadoop-common in the patch passed.
          +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          -1 unit 8m 15s hadoop-common in the patch failed.
          +1 unit 1m 37s hadoop-azure in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          91m 2s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853922/HADOOP-13930.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c8016ee3d853 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f1c9caf
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 13m 37s trunk passed +1 compile 14m 3s trunk passed +1 checkstyle 1m 56s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 0s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 57s the patch passed +1 compile 11m 30s the patch passed +1 javac 11m 30s the patch passed +1 checkstyle 1m 57s the patch passed +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 2m 20s the patch passed +1 javadoc 0m 53s hadoop-common in the patch passed. +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) -1 unit 8m 15s hadoop-common in the patch failed. +1 unit 1m 37s hadoop-azure in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 91m 2s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853922/HADOOP-13930.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c8016ee3d853 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f1c9caf Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11687/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          Hopefully this should fix KerberosAuthException thrown from tests in hadoop-common

          testFileOutput(org.apache.hadoop.security.TestKDiag) Time elapsed: 0.039 sec <<< ERROR!
          org.apache.hadoop.security.KerberosAuthException: Login failure for user: foo@EXAMPLE.COM from keytab /testptch/hadoop/hadoop-common-project/hadoop-common/target/keytab javax.security.auth.login.LoginException: Unable to obtain password from user

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - Hopefully this should fix KerberosAuthException thrown from tests in hadoop-common testFileOutput(org.apache.hadoop.security.TestKDiag) Time elapsed: 0.039 sec <<< ERROR! org.apache.hadoop.security.KerberosAuthException: Login failure for user: foo@EXAMPLE.COM from keytab /testptch/hadoop/hadoop-common-project/hadoop-common/target/keytab javax.security.auth.login.LoginException: Unable to obtain password from user
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          0 mvndep 0m 20s Maven dependency ordering for branch
          +1 mvninstall 12m 41s trunk passed
          +1 compile 12m 54s trunk passed
          +1 checkstyle 1m 52s trunk passed
          +1 mvnsite 1m 26s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 7s trunk passed
          +1 javadoc 1m 15s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 3s the patch passed
          +1 compile 12m 7s the patch passed
          +1 javac 12m 7s the patch passed
          +1 checkstyle 2m 5s the patch passed
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 findbugs 2m 28s the patch passed
          +1 javadoc 0m 54s hadoop-common in the patch passed.
          +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          -1 unit 8m 26s hadoop-common in the patch failed.
          +1 unit 1m 40s hadoop-azure in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          90m 9s



          Reason Tests
          Failed junit tests hadoop.security.TestKDiag



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853910/HADOOP-13930.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 4055fca83bb4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f1c9caf
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 12m 41s trunk passed +1 compile 12m 54s trunk passed +1 checkstyle 1m 52s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 7s trunk passed +1 javadoc 1m 15s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 12m 7s the patch passed +1 javac 12m 7s the patch passed +1 checkstyle 2m 5s the patch passed +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 2m 28s the patch passed +1 javadoc 0m 54s hadoop-common in the patch passed. +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) -1 unit 8m 26s hadoop-common in the patch failed. +1 unit 1m 40s hadoop-azure in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 90m 9s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853910/HADOOP-13930.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 4055fca83bb4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f1c9caf Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11686/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          Fixed hadoop-common unit-test

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - Fixed hadoop-common unit-test
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 2m 19s Maven dependency ordering for branch
          +1 mvninstall 14m 55s trunk passed
          +1 compile 14m 34s trunk passed
          +1 checkstyle 2m 4s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 1m 11s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 0s the patch passed
          +1 compile 11m 48s the patch passed
          +1 javac 11m 48s the patch passed
          +1 checkstyle 1m 54s the patch passed
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 findbugs 2m 24s the patch passed
          +1 javadoc 0m 54s hadoop-common in the patch passed.
          +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          -1 unit 8m 5s hadoop-common in the patch failed.
          +1 unit 2m 10s hadoop-azure in the patch passed.
          +1 asflicense 0m 39s The patch does not generate ASF License warnings.
          95m 58s



          Reason Tests
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853872/HADOOP-13930.007.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 98976a3e7900 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 003ae00
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 2m 19s Maven dependency ordering for branch +1 mvninstall 14m 55s trunk passed +1 compile 14m 34s trunk passed +1 checkstyle 2m 4s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 0s the patch passed +1 compile 11m 48s the patch passed +1 javac 11m 48s the patch passed +1 checkstyle 1m 54s the patch passed +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 findbugs 2m 24s the patch passed +1 javadoc 0m 54s hadoop-common in the patch passed. +1 javadoc 0m 24s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) -1 unit 8m 5s hadoop-common in the patch failed. +1 unit 2m 10s hadoop-azure in the patch passed. +1 asflicense 0m 39s The patch does not generate ASF License warnings. 95m 58s Reason Tests Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853872/HADOOP-13930.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 98976a3e7900 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 003ae00 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11685/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          This patch has the same content as 006.patch. The automated build did not start for some reason when 006.patch was submitted. Trying again by renaming as 007.patch.

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - This patch has the same content as 006.patch. The automated build did not start for some reason when 006.patch was submitted. Trying again by renaming as 007.patch.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          HADOOP-13930.006.patch attached yesterday did not start the automated build somehow. Attaching the same patch as 007.patch.

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - HADOOP-13930 .006.patch attached yesterday did not start the automated build somehow. Attaching the same patch as 007.patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 12m 30s trunk passed
          +1 compile 12m 51s trunk passed
          +1 checkstyle 1m 53s trunk passed
          +1 mvnsite 1m 27s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 10s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 0m 58s the patch passed
          +1 compile 11m 39s the patch passed
          +1 javac 11m 39s the patch passed
          +1 checkstyle 1m 56s the patch passed
          +1 mvnsite 1m 30s the patch passed
          +1 mvneclipse 0m 44s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          -1 whitespace 0m 0s The patch 1 line(s) with tabs.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          +1 findbugs 2m 21s the patch passed
          +1 javadoc 0m 52s hadoop-common in the patch passed.
          +1 javadoc 0m 25s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          -1 unit 8m 13s hadoop-common in the patch failed.
          +1 unit 1m 38s hadoop-azure in the patch passed.
          +1 asflicense 0m 38s The patch does not generate ASF License warnings.
          88m 57s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin
            hadoop.security.TestKDiag
            hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853695/HADOOP-13930.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux d825485de7ec 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6ba61d2
          Default Java 1.8.0_121
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/whitespace-tabs.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 12m 30s trunk passed +1 compile 12m 51s trunk passed +1 checkstyle 1m 53s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 10s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 11m 39s the patch passed +1 javac 11m 39s the patch passed +1 checkstyle 1m 56s the patch passed +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 44s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 0s The patch 1 line(s) with tabs. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 2m 21s the patch passed +1 javadoc 0m 52s hadoop-common in the patch passed. +1 javadoc 0m 25s hadoop-tools_hadoop-azure generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) -1 unit 8m 13s hadoop-common in the patch failed. +1 unit 1m 38s hadoop-azure in the patch passed. +1 asflicense 0m 38s The patch does not generate ASF License warnings. 88m 57s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.security.TestKDiag   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853695/HADOOP-13930.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux d825485de7ec 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6ba61d2 Default Java 1.8.0_121 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11671/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853675/HADOOP-13930.004.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11670/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853675/HADOOP-13930.004.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11670/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853672/HADOOP-13930.003.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11669/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12853672/HADOOP-13930.003.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11669/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          Hi Mingliang Liu,Steve Loughran.

          I have attached a new patch HADOOP-13930.003.patch that addresses Steve's review comments.
          Could you please review at your earliest ?

          Thanks
          Siva

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - Hi Mingliang Liu , Steve Loughran . I have attached a new patch HADOOP-13930 .003.patch that addresses Steve's review comments. Could you please review at your earliest ? Thanks Siva
          Hide
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment -

          Fixes for review comments from Steve Loughran on 29/Dec/16 16:23

          Show
          sisankar@microsoft.com Sivaguru Sankaridurg added a comment - Fixes for review comments from Steve Loughran on 29/Dec/16 16:23
          Hide
          liuml07 Mingliang Liu added a comment -

          Assigning to Sivaguru Sankaridurg.

          Show
          liuml07 Mingliang Liu added a comment - Assigning to Sivaguru Sankaridurg .
          Hide
          liuml07 Mingliang Liu added a comment -

          Hi Dushyanth, do you have updated patch available that addresses all Steve's comments? I'd like to get this in trunk and branch-2. Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Hi Dushyanth , do you have updated patch available that addresses all Steve's comments? I'd like to get this in trunk and branch-2 . Thanks,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 49s Maven dependency ordering for branch
          +1 mvninstall 12m 36s trunk passed
          +1 compile 9m 43s trunk passed
          +1 checkstyle 1m 36s trunk passed
          +1 mvnsite 1m 28s trunk passed
          +1 mvneclipse 0m 38s trunk passed
          +1 findbugs 2m 3s trunk passed
          +1 javadoc 3m 23s trunk passed
          0 mvndep 0m 53s Maven dependency ordering for patch
          +1 mvninstall 3m 5s the patch passed
          +1 compile 13m 26s the patch passed
          +1 javac 13m 26s the patch passed
          -0 checkstyle 1m 34s root: The patch generated 9 new + 14 unchanged - 0 fixed = 23 total (was 14)
          +1 mvnsite 1m 28s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 0m 46s hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 23s hadoop-azure in the patch failed.
          -1 unit 7m 44s hadoop-common in the patch failed.
          +1 unit 1m 36s hadoop-azure in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          93m 33s



          Reason Tests
          FindBugs module:hadoop-tools/hadoop-azure
            org.apache.hadoop.fs.azure.NativeAzureFileSystem.KEY_AZURE_AUTHORIZATION isn't final but should be At NativeAzureFileSystem.java:be At NativeAzureFileSystem.java:[line 1103]
          Failed junit tests hadoop.conf.TestCommonConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845516/HADOOP-13930.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux af20db2062b9 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e49e0a6
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 49s Maven dependency ordering for branch +1 mvninstall 12m 36s trunk passed +1 compile 9m 43s trunk passed +1 checkstyle 1m 36s trunk passed +1 mvnsite 1m 28s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 3s trunk passed +1 javadoc 3m 23s trunk passed 0 mvndep 0m 53s Maven dependency ordering for patch +1 mvninstall 3m 5s the patch passed +1 compile 13m 26s the patch passed +1 javac 13m 26s the patch passed -0 checkstyle 1m 34s root: The patch generated 9 new + 14 unchanged - 0 fixed = 23 total (was 14) +1 mvnsite 1m 28s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 0m 46s hadoop-tools/hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 23s hadoop-azure in the patch failed. -1 unit 7m 44s hadoop-common in the patch failed. +1 unit 1m 36s hadoop-azure in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 93m 33s Reason Tests FindBugs module:hadoop-tools/hadoop-azure   org.apache.hadoop.fs.azure.NativeAzureFileSystem.KEY_AZURE_AUTHORIZATION isn't final but should be At NativeAzureFileSystem.java:be At NativeAzureFileSystem.java: [line 1103] Failed junit tests hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845516/HADOOP-13930.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux af20db2062b9 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e49e0a6 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/new-findbugs-hadoop-tools_hadoop-azure.html javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11354/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          dchickabasapa Dushyanth added a comment -

          Adding re-based patch and addressing some of Steve's comment. Will address test related comments in the next patch.

          Show
          dchickabasapa Dushyanth added a comment - Adding re-based patch and addressing some of Steve's comment. Will address test related comments in the next patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13930
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844284/HADOOP-13930.001.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11348/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-13930 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844284/HADOOP-13930.001.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11348/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          initial review

          • I'm not sure about the name fs.azure.enable.authorization. I'd prefer something like fs.azure.authorization *for all azure auth properties. This would include "fs.azure.remote.auth.service.url", as an example
          • NativeAzureFileSystem auth checks all look like cut & paste code. Better to have some method "void checkAuth(String operation) throws IOE {}" which is a no-op if auth isn't needed; checks and rejects if not.
          • RemoteWasbAuthorizerImpl. That exception handler should just be a single multipart catch(URISyntaxException | WasbRemoteCallException ...) handler
          • and as the WasB exceptions should all be IOEs, don't bother catch/wrap/rethrowing those
          • Auth interface methods should declare the right to raise IOEs, not just WasbAuthorizationException,

          WasbRemoteCallHelper

          • check content type matches expected. proxies getting in the way could serve up TEXT/PLAIN and that needs to be rejected with a meaningful message, rather than an json parse exception.
          • look at content length, recognise & react to not enough content. As it is, an endpoint set to serve up an infinite amount of data on a GET Would overload the client heap, trivially
          • methods to declare throws IOE.
          • do include URL at fault in all exception messages. Your support team will appreciate this.

          WasbAuthorizerInterface

          What does it mean if init fails? Better: make void return type, add throws IOE; allow implementations to do this

          Testing

          • Please use ContractTestUtils for all assertions about FS state; it includes diagnostics on problems.
          • Your tests need to use paths which are unique for the test suite; using the working path doesn't do that. We need this for parallel test execution
          • test path must be robustly deleted in teardown; some appear to create, only one deletes. I know the other tests are expected to fail, but if they do create the path, there's trouble in the FS.
          • testRenameAccessCheckNegative: why catch/rethrow the exception?
          • testReadAccessCheckPositive(). In ContractTestUtils there's something to read data off a file.
          • No tests for RemoteWasbAuthorizerImpl. A mock remote WasbRemoteCallHelper could enable that; serve up different JSON docs, including invalid ones, HTTP errors, etc.
          Show
          stevel@apache.org Steve Loughran added a comment - initial review I'm not sure about the name fs.azure.enable.authorization . I'd prefer something like fs.azure.authorization *for all azure auth properties. This would include "fs.azure.remote.auth.service.url" , as an example NativeAzureFileSystem auth checks all look like cut & paste code. Better to have some method "void checkAuth(String operation) throws IOE {}" which is a no-op if auth isn't needed; checks and rejects if not. RemoteWasbAuthorizerImpl . That exception handler should just be a single multipart catch(URISyntaxException | WasbRemoteCallException ...) handler and as the WasB exceptions should all be IOEs, don't bother catch/wrap/rethrowing those Auth interface methods should declare the right to raise IOEs, not just WasbAuthorizationException, WasbRemoteCallHelper check content type matches expected. proxies getting in the way could serve up TEXT/PLAIN and that needs to be rejected with a meaningful message, rather than an json parse exception. look at content length, recognise & react to not enough content. As it is, an endpoint set to serve up an infinite amount of data on a GET Would overload the client heap, trivially methods to declare throws IOE. do include URL at fault in all exception messages. Your support team will appreciate this. WasbAuthorizerInterface What does it mean if init fails? Better: make void return type, add throws IOE ; allow implementations to do this Testing Please use ContractTestUtils for all assertions about FS state; it includes diagnostics on problems. Your tests need to use paths which are unique for the test suite; using the working path doesn't do that. We need this for parallel test execution test path must be robustly deleted in teardown; some appear to create, only one deletes. I know the other tests are expected to fail, but if they do create the path, there's trouble in the FS. testRenameAccessCheckNegative : why catch/rethrow the exception? testReadAccessCheckPositive() . In ContractTestUtils there's something to read data off a file. No tests for RemoteWasbAuthorizerImpl . A mock remote WasbRemoteCallHelper could enable that; serve up different JSON docs, including invalid ones, HTTP errors, etc.
          Hide
          dchickabasapa Dushyanth added a comment -

          Attaching first patch. This patch is using WasbRemoteCallHelper class that is yet to be committed via HADOOP-13863.

          The patch has been tested against Live Azure storage tests and also newly added tests for the feature.

          Show
          dchickabasapa Dushyanth added a comment - Attaching first patch. This patch is using WasbRemoteCallHelper class that is yet to be committed via HADOOP-13863 . The patch has been tested against Live Azure storage tests and also newly added tests for the feature.

            People

            • Assignee:
              sisankar@microsoft.com Sivaguru Sankaridurg
              Reporter:
              dchickabasapa Dushyanth
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development