Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14443

Azure: Support retry and client side failover for authorization, SASKey and delegation token generation

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fs/azure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, WasRemoteCallHelper can be configured to talk to only one URL for authorization, SASKey generation and delegation token generation. If for some reason the service is down, all the requests will fail.
      So proposal is to,

      • Add support to configure multiple URLs, so that if communication to one URL fails, client can retry on another instance of the service running on different node for authorization, SASKey generation and delegation token generation.
      • Rename the configurations fs.azure.authorization.remote.service.url to fs.azure.authorization.remote.service.urls and fs.azure.cred.service.url to fs.azure.cred.service.urls to support the comma separated list of URLs.
      • Introduce a new configuration fs.azure.delegation.token.service.urls to configure the comma separated list of service URLs to get the delegation token.
      1. HADOOP-14443.1.patch
        89 kB
        Santhosh G Nayak
      2. HADOOP-14443.2.patch
        97 kB
        Santhosh G Nayak
      3. HADOOP-14443.3.patch
        97 kB
        Santhosh G Nayak
      4. HADOOP-14443.4.patch
        97 kB
        Santhosh G Nayak
      5. HADOOP-14443.5.patch
        101 kB
        Santhosh G Nayak
      6. HADOOP-14443.6.patch
        102 kB
        Mingliang Liu
      7. HADOOP-14443.7.patch
        102 kB
        Mingliang Liu
      8. HADOOP-14443-branch-2.2.patch
        101 kB
        Jitendra Nath Pandey
      9. HADOOP-14443-branch2-1.patch
        100 kB
        Santhosh G Nayak
      10. HADOOP-14443-branch2-2.patch
        101 kB
        Santhosh G Nayak

        Issue Links

          Activity

          Hide
          jnp Jitendra Nath Pandey added a comment -

          Committed to branch-2. Thanks to Santhosh G Nayak.

          Show
          jnp Jitendra Nath Pandey added a comment - Committed to branch-2. Thanks to Santhosh G Nayak.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          I will commit this to branch-2 shortly.

          Show
          jnp Jitendra Nath Pandey added a comment - I will commit this to branch-2 shortly.
          Hide
          anu Anu Engineer added a comment -

          I have ran test-patch on my local machine and verified that this patch works correctly on branch 2. +1, for committing, seems like we are having some issue with jenkins and branch-2

          Show
          anu Anu Engineer added a comment - I have ran test-patch on my local machine and verified that this patch works correctly on branch 2. +1, for committing, seems like we are having some issue with jenkins and branch-2
          Show
          anu Anu Engineer added a comment - - edited I have started a pre-commit build here. https://builds.apache.org/blue/organizations/jenkins/Hadoop-branch2-parameterized/detail/Hadoop-branch2-parameterized/23/pipeline
          Hide
          jnp Jitendra Nath Pandey added a comment -

          +1 for the latest branch-2 patch. Need a jenkins run before commit.

          Renamed the last branch-2 patch for jenkins to pick up.

          Show
          jnp Jitendra Nath Pandey added a comment - +1 for the latest branch-2 patch. Need a jenkins run before commit. Renamed the last branch-2 patch for jenkins to pick up.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-14443 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876126/HADOOP-14443-branch2-2.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12735/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-14443 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876126/HADOOP-14443-branch2-2.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12735/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Thanks Mingliang Liu, I missed to run it in secure mode. Attached a new patch with the fix.

          Show
          snayak Santhosh G Nayak added a comment - Thanks Mingliang Liu , I missed to run it in secure mode. Attached a new patch with the fix.
          Hide
          liuml07 Mingliang Liu added a comment -

          I ran the tests for the branch-2 patch, and found the following errors.

          Failed tests:
            TestWasbRemoteCallHelper.testMalFormedJSONResponse
          Expected: (an instance of org.apache.hadoop.fs.azure.WasbAuthorizationException and exception with message a string containing "com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input in FIELD_NAME")
               but: an instance of org.apache.hadoop.fs.azure.WasbAuthorizationException <org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries
           at [Source: java.io.StringReader@6b3e12b5; line: 1, column: 131]> is a org.codehaus.jackson.JsonParseException
          Stacktrace was: org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries
           at [Source: java.io.StringReader@6b3e12b5; line: 1, column: 131]
          	at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
          	at org.codehaus.jackson.impl.ReaderBasedParser._skipWS(ReaderBasedParser.java:1470)
          	at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:425)
          	at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:690)
          	at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580)
          	at org.codehaus.jackson.map.ObjectReader._bindAndClose(ObjectReader.java:768)
          	at org.codehaus.jackson.map.ObjectReader.readValue(ObjectReader.java:460)
          	at org.apache.hadoop.fs.azure.RemoteWasbAuthorizerImpl.authorize(RemoteWasbAuthorizerImpl.java:153)
          	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.performAuthCheck(NativeAzureFileSystem.java:1468)
          	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1704)
          	at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1554)
          	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1067)
          	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1048)
          	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:937)
          	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:925)
          	at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.performop(TestWasbRemoteCallHelper.java:445)
          	at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testMalFormedJSONResponse(TestWasbRemoteCallHelper.java:247)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:497)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
          	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
          	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
          	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
          	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
          	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
          	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
          	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
          	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
          	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
          	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
          

          Can you fix that on branch-2? Thanks,

          Show
          liuml07 Mingliang Liu added a comment - I ran the tests for the branch-2 patch, and found the following errors. Failed tests: TestWasbRemoteCallHelper.testMalFormedJSONResponse Expected: (an instance of org.apache.hadoop.fs.azure.WasbAuthorizationException and exception with message a string containing "com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input in FIELD_NAME" ) but: an instance of org.apache.hadoop.fs.azure.WasbAuthorizationException <org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries at [Source: java.io.StringReader@6b3e12b5; line: 1, column: 131]> is a org.codehaus.jackson.JsonParseException Stacktrace was: org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries at [Source: java.io.StringReader@6b3e12b5; line: 1, column: 131] at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433) at org.codehaus.jackson.impl.ReaderBasedParser._skipWS(ReaderBasedParser.java:1470) at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:425) at org.codehaus.jackson.map.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:690) at org.codehaus.jackson.map.deser.BeanDeserializer.deserialize(BeanDeserializer.java:580) at org.codehaus.jackson.map.ObjectReader._bindAndClose(ObjectReader.java:768) at org.codehaus.jackson.map.ObjectReader.readValue(ObjectReader.java:460) at org.apache.hadoop.fs.azure.RemoteWasbAuthorizerImpl.authorize(RemoteWasbAuthorizerImpl.java:153) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.performAuthCheck(NativeAzureFileSystem.java:1468) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1704) at org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1554) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1067) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1048) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:937) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:925) at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.performop(TestWasbRemoteCallHelper.java:445) at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testMalFormedJSONResponse(TestWasbRemoteCallHelper.java:247) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) Can you fix that on branch-2? Thanks,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-14443 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875503/HADOOP-14443-branch2-1.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12704/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-14443 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875503/HADOOP-14443-branch2-1.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12704/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Thanks Mingliang Liu for committing the patch to trunk. Attached a patch for branch-2 with ObjectMapper optimization and resolved conflicts. Could you please review and commit it?

          Show
          snayak Santhosh G Nayak added a comment - Thanks Mingliang Liu for committing the patch to trunk . Attached a patch for branch-2 with ObjectMapper optimization and resolved conflicts. Could you please review and commit it?
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11959 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11959/)
          HADOOP-14443. Azure: Support retry and client side failover for (liuml07: rev 38996fdcf0987d1da00ce46f8284d8fcdce57329)

          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/JsonUtils.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenManager.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java
          • (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java
          • (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/SecurityUtils.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java
          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java
          • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureWasbRemoteCallHelper.java
          • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java
          • (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/RemoteWasbDelegationTokenManager.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11959 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11959/ ) HADOOP-14443 . Azure: Support retry and client side failover for (liuml07: rev 38996fdcf0987d1da00ce46f8284d8fcdce57329) (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/JsonUtils.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbDelegationTokenManager.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteWasbAuthorizerImpl.java (edit) hadoop-tools/hadoop-azure/src/site/markdown/index.md (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/Constants.java (delete) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/SecurityUtils.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/NativeAzureFileSystem.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/RemoteSASKeyGeneratorImpl.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/WasbTokenRenewer.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestWasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestNativeAzureFileSystemAuthorization.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SecureWasbRemoteCallHelper.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/WasbRemoteCallHelper.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/TokenUtils.java (add) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/security/RemoteWasbDelegationTokenManager.java
          Hide
          liuml07 Mingliang Liu added a comment -

          Committed to trunk branch.

          Thanks for your contribution, Santhosh G Nayak. Thanks for your review Steve Loughran.

          When I backport to branch-2, I found major conflicts. This is because ObjectReader optimization is not in branch-2. Can you upload a branch-2 patch? Thanks,

          Show
          liuml07 Mingliang Liu added a comment - Committed to trunk branch. Thanks for your contribution, Santhosh G Nayak . Thanks for your review Steve Loughran . When I backport to branch-2, I found major conflicts. This is because ObjectReader optimization is not in branch-2. Can you upload a branch-2 patch? Thanks,
          Hide
          snayak Santhosh G Nayak added a comment - - edited

          Thanks Mingliang Liu. v7 version of the patch looks good. Tested against Azure Storage South India endpoint. Following is the test report,

          PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.451 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.47 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.432 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.42 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.524 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.378 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.448 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.589 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
          Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.732 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
          Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
          Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.403 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
          Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
          Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.731 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
          Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.885 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
          Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.62 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
          Running org.apache.hadoop.fs.azure.TestBlobDataValidation
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.685 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation
          Running org.apache.hadoop.fs.azure.TestBlobMetadata
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.24 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata
          Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.167 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
          Running org.apache.hadoop.fs.azure.TestContainerChecks
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.588 sec - in org.apache.hadoop.fs.azure.TestContainerChecks
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 378.382 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.937 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
          Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 425.129 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 130.504 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.671 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
          Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 7.166 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
          Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 5.773 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.291 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.518 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.771 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.408 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.717 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.625 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.365 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 166.31 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.187 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 452.95 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.95 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
          Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.394 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.085 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
          Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 428.76 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
          Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.566 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
          Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.849 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
          Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 173.356 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
          Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
          Running org.apache.hadoop.fs.azure.TestWasbFsck
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.066 sec - in org.apache.hadoop.fs.azure.TestWasbFsck
          Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.896 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
          Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 14.022 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
          
          Results :
          
          Tests run: 707, Failures: 0, Errors: 0, Skipped: 203
          
          Show
          snayak Santhosh G Nayak added a comment - - edited Thanks Mingliang Liu . v7 version of the patch looks good. Tested against Azure Storage South India endpoint. Following is the test report, PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.451 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.47 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.432 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.42 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.524 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.378 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.448 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.589 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.732 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.403 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.263 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 31.731 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.885 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.62 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Running org.apache.hadoop.fs.azure.TestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.685 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation Running org.apache.hadoop.fs.azure.TestBlobMetadata Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.24 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.167 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Running org.apache.hadoop.fs.azure.TestContainerChecks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.588 sec - in org.apache.hadoop.fs.azure.TestContainerChecks Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.563 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 378.382 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 41.937 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 425.129 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 130.504 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.671 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 7.166 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 5.773 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.291 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.518 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.771 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.408 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.717 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 149.625 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 2.365 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 166.31 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.187 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 452.95 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 17.95 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.394 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.085 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 428.76 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.566 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.849 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 173.356 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Running org.apache.hadoop.fs.azure.TestWasbFsck Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.066 sec - in org.apache.hadoop.fs.azure.TestWasbFsck Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.896 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 14.022 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Results : Tests run: 707, Failures: 0, Errors: 0, Skipped: 203
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 59s trunk passed
          +1 compile 0m 20s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 18s the patch passed
          +1 javac 0m 18s the patch passed
          +1 checkstyle 0m 12s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50)
          +1 mvnsite 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 37s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 1m 30s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          22m 2s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875184/HADOOP-14443.7.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c157f9bd39a7 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3be2659
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12683/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12683/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 59s trunk passed +1 compile 0m 20s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 18s the patch passed +1 javac 0m 18s the patch passed +1 checkstyle 0m 12s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50) +1 mvnsite 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 37s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 1m 30s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 22m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875184/HADOOP-14443.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c157f9bd39a7 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3be2659 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12683/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12683/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          V7 is only to make checkstyle clear.

          Show
          liuml07 Mingliang Liu added a comment - V7 is only to make checkstyle clear.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 12m 39s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 17s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 27 unchanged - 23 fixed = 29 total (was 50)
          +1 mvnsite 0m 19s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 31s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 1m 28s hadoop-azure in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          19m 23s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875176/HADOOP-14443.6.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 01d7214e3394 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3be2659
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 12m 39s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 2 new + 27 unchanged - 23 fixed = 29 total (was 50) +1 mvnsite 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 31s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 28s hadoop-azure in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 19m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875176/HADOOP-14443.6.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 01d7214e3394 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3be2659 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12682/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          liuml07 Mingliang Liu added a comment -

          V6 patch:

          • Rebase from trunk, resolving conflicts
          • Fix some unreported checkstyle
          • Refactor checkHttpGetMatchHost helper method in test to reuse code
          • Tested with US West region.

          I'm happy with it now. I'll commit if it looks good to you Santhosh G Nayak (if you can run test again, that will be great).

          Show
          liuml07 Mingliang Liu added a comment - V6 patch: Rebase from trunk, resolving conflicts Fix some unreported checkstyle Refactor checkHttpGetMatchHost helper method in test to reuse code Tested with US West region. I'm happy with it now. I'll commit if it looks good to you Santhosh G Nayak (if you can run test again, that will be great).
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 4s trunk passed
          +1 compile 0m 19s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 21s trunk passed
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          -0 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 1 new + 27 unchanged - 23 fixed = 28 total (was 50)
          +1 mvnsite 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 32s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 1m 20s hadoop-azure in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          19m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875039/HADOOP-14443.5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 90d6a6b7a0bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ea1da39
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 4s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 21s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed -0 checkstyle 0m 10s hadoop-tools/hadoop-azure: The patch generated 1 new + 27 unchanged - 23 fixed = 28 total (was 50) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 1m 20s hadoop-azure in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 19m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12875039/HADOOP-14443.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 90d6a6b7a0bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ea1da39 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12660/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Thanks Mingliang Liu for reviewing v4 version of the patch. Attaching v5 version of the patch with following addressed,
          1. Updated index.md with recent changes.
          2. Made changes to reuse ObjectMapper in JsonUtils#parse().
          3. Tested it with South India region of Azure storage endpoint.

          PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          
          -------------------------------------------------------
           T E S T S
          -------------------------------------------------------
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
          Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.508 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
          Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.432 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.593 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.402 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.546 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.409 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.417 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename
          Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.564 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek
          Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
          Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.205 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation
          Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.659 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater
          Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem
          Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.246 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage
          Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.164 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo
          Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.045 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode
          Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.34 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions
          Running org.apache.hadoop.fs.azure.TestBlobDataValidation
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.019 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation
          Running org.apache.hadoop.fs.azure.TestBlobMetadata
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.242 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata
          Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.301 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference
          Running org.apache.hadoop.fs.azure.TestContainerChecks
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.602 sec - in org.apache.hadoop.fs.azure.TestContainerChecks
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.632 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
          Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 472.856 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.131 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded
          Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
          Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 529.75 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.019 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.787 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
          Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 6.296 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
          Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 6.661 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.194 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.576 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.691 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.863 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.715 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 167.427 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.998 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
          Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 166.18 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
          Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 474.237 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.112 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
          Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.24 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
          Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
          Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.069 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic
          Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
          Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 503.22 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive
          Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
          Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.199 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations
          Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
          Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.845 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive
          Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
          Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 230.55 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite
          Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.37 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider
          Running org.apache.hadoop.fs.azure.TestWasbFsck
          Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.091 sec - in org.apache.hadoop.fs.azure.TestWasbFsck
          Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
          Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 3.752 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
          Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
          Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 13.123 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration
          
          Results :
          
          Tests run: 707, Failures: 0, Errors: 0, Skipped: 203
          
          Show
          snayak Santhosh G Nayak added a comment - Thanks Mingliang Liu for reviewing v4 version of the patch. Attaching v5 version of the patch with following addressed, 1. Updated index.md with recent changes. 2. Made changes to reuse ObjectMapper in JsonUtils#parse() . 3. Tested it with South India region of Azure storage endpoint. PS D:\code\hadoop1\hadoop\hadoop-tools\hadoop-azure> mvn test -q ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Tests run: 5, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 0.508 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractAppend Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Tests run: 11, Failures: 0, Errors: 0, Skipped: 11, Time elapsed: 0.432 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractCreate Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Tests run: 8, Failures: 0, Errors: 0, Skipped: 8, Time elapsed: 0.593 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDelete Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.402 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractDistCp Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.546 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractGetFileStatus Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Tests run: 7, Failures: 0, Errors: 0, Skipped: 7, Time elapsed: 0.409 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractMkdir Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.417 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractOpen Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.385 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractRename Running org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Tests run: 18, Failures: 0, Errors: 0, Skipped: 18, Time elapsed: 0.564 sec - in org.apache.hadoop.fs.azure.contract.TestAzureNativeContractSeek Running org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 78.205 sec - in org.apache.hadoop.fs.azure.metrics.TestAzureFileSystemInstrumentation Running org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.659 sec - in org.apache.hadoop.fs.azure.metrics.TestBandwidthGaugeUpdater Running org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.288 sec - in org.apache.hadoop.fs.azure.metrics.TestNativeAzureFileSystemMetricsSystem Running org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.246 sec - in org.apache.hadoop.fs.azure.metrics.TestRollingWindowAverage Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.164 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo Running org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.045 sec - in org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIoWithSecureMode Running org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.34 sec - in org.apache.hadoop.fs.azure.TestAzureFileSystemErrorConditions Running org.apache.hadoop.fs.azure.TestBlobDataValidation Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.019 sec - in org.apache.hadoop.fs.azure.TestBlobDataValidation Running org.apache.hadoop.fs.azure.TestBlobMetadata Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.242 sec - in org.apache.hadoop.fs.azure.TestBlobMetadata Running org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.301 sec - in org.apache.hadoop.fs.azure.TestBlobTypeSpeedDifference Running org.apache.hadoop.fs.azure.TestContainerChecks Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.602 sec - in org.apache.hadoop.fs.azure.TestContainerChecks Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.632 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionHandling Running org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Tests run: 47, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 472.856 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationExceptionMessage Running org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 55.131 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsExceptionHandlingMultiThreaded Running org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 529.75 sec - in org.apache.hadoop.fs.azure.TestFileSystemOperationsWithThreads Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.019 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAppend Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.787 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAtomicRenameDirList Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Tests run: 21, Failures: 0, Errors: 0, Skipped: 21, Time elapsed: 6.296 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorization Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Tests run: 24, Failures: 0, Errors: 0, Skipped: 24, Time elapsed: 6.661 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemAuthorizationWithOwner Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.194 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemBlockLocations Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.576 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemClientLogging Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.691 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrency Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.863 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemConcurrencyLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Tests run: 43, Failures: 0, Errors: 0, Skipped: 43, Time elapsed: 0.715 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractEmulator Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 167.427 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 1.998 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Tests run: 43, Failures: 0, Errors: 0, Skipped: 5, Time elapsed: 166.18 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemContractPageBlobLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.277 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Tests run: 51, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 474.237 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemLive Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 18.112 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Tests run: 50, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 2.24 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked Running org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Tests run: 3, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 0.069 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFileSystemUploadLogic Running org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Tests run: 46, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 503.22 sec - in org.apache.hadoop.fs.azure.TestNativeAzureFSPageBlobLive Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.199 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperations Running org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.845 sec - in org.apache.hadoop.fs.azure.TestOutOfBandAzureBlobOperationsLive Running org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 230.55 sec - in org.apache.hadoop.fs.azure.TestReadAndSeekPageBlobAfterWrite Running org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.37 sec - in org.apache.hadoop.fs.azure.TestShellDecryptionKeyProvider Running org.apache.hadoop.fs.azure.TestWasbFsck Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 1.091 sec - in org.apache.hadoop.fs.azure.TestWasbFsck Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 3.752 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Running org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Tests run: 18, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 13.123 sec - in org.apache.hadoop.fs.azure.TestWasbUriAndConfiguration Results : Tests run: 707, Failures: 0, Errors: 0, Skipped: 203
          Hide
          liuml07 Mingliang Liu added a comment -

          Santhosh G Nayak, thanks for updating the patch.

          The current v4 version looks good to me overall. I have a few minor comments before commit. I can provide more input this week for another round review.

          1. Do we need to change related documentations about this? e.g. index.md, core-default.xml etc
          2. JsonUtils#parse() should reuse the ObjectMapper, see HADOOP-14601
          3. Can you post test report with azure web service?
          Show
          liuml07 Mingliang Liu added a comment - Santhosh G Nayak , thanks for updating the patch. The current v4 version looks good to me overall. I have a few minor comments before commit. I can provide more input this week for another round review. Do we need to change related documentations about this? e.g. index.md, core-default.xml etc JsonUtils#parse() should reuse the ObjectMapper , see HADOOP-14601 Can you post test report with azure web service?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 19s trunk passed
          +1 compile 0m 18s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 20s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 18s the patch passed
          +1 compile 0m 16s the patch passed
          +1 javac 0m 16s the patch passed
          +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50)
          +1 mvnsite 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 1m 18s hadoop-azure in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          19m 45s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872709/HADOOP-14443.4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c4ffdac51068 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d64c842
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 19s trunk passed +1 compile 0m 18s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 20s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed +1 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 1m 18s hadoop-azure in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 19m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872709/HADOOP-14443.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c4ffdac51068 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d64c842 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12520/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Attaching HADOOP-14443.4.patch to fix javac, Javadoc subsystem related failures.

          Show
          snayak Santhosh G Nayak added a comment - Attaching HADOOP-14443 .4.patch to fix javac , Javadoc subsystem related failures.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 13m 54s trunk passed
          +1 compile 0m 22s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 findbugs 0m 31s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 18s the patch passed
          -1 javac 0m 18s hadoop-tools_hadoop-azure generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
          +1 checkstyle 0m 13s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50)
          +1 mvnsite 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 38s the patch passed
          -1 javadoc 0m 12s hadoop-azure in the patch failed.
          +1 unit 1m 27s hadoop-azure in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          21m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872671/HADOOP-14443.3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d2e30b1c08b3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e86eef9
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 13m 54s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 22s trunk passed +1 findbugs 0m 31s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 18s the patch passed -1 javac 0m 18s hadoop-tools_hadoop-azure generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 checkstyle 0m 13s hadoop-tools/hadoop-azure: The patch generated 0 new + 27 unchanged - 23 fixed = 27 total (was 50) +1 mvnsite 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 38s the patch passed -1 javadoc 0m 12s hadoop-azure in the patch failed. +1 unit 1m 27s hadoop-azure in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 21m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12872671/HADOOP-14443.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d2e30b1c08b3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e86eef9 Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12519/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Mingliang Liu Thanks for the comments. I have fixed few of those and attached a new patch,
          1. @Override annotation to new line.
          2. Marked the parameters such as op, renewer, service and token as constants?
          3. There is a default retry spec which will be used, if user wants to override, he can do so. If this is not recommended in Hadoop codebase, we can remove the configuration.
          4. Added Thread.currentThread().interrupt() in WasbRemoteCallHelper::shouldRetry() when InterruptedException occurs.
          5. This patch consolidates various http helpers which make common SPNEGO connection into WasbRemoteCallHelper and SecureWasbRemoteCallHelper. It makes easier to apply the same retry/failover logic to various remote HTTP calls. I am afraid separating it may be difficult.

          Show
          snayak Santhosh G Nayak added a comment - Mingliang Liu Thanks for the comments. I have fixed few of those and attached a new patch, 1. @Override annotation to new line. 2. Marked the parameters such as op , renewer , service and token as constants? 3. There is a default retry spec which will be used, if user wants to override, he can do so. If this is not recommended in Hadoop codebase, we can remove the configuration. 4. Added Thread.currentThread().interrupt() in WasbRemoteCallHelper::shouldRetry() when InterruptedException occurs. 5. This patch consolidates various http helpers which make common SPNEGO connection into WasbRemoteCallHelper and SecureWasbRemoteCallHelper . It makes easier to apply the same retry/failover logic to various remote HTTP calls. I am afraid separating it may be difficult.
          Hide
          liuml07 Mingliang Liu added a comment -

          I will review this one shortly.

          Early minor comments:

          1. Can you make the @Override annotation in a separate line?
          2. The URI build code like following, can we make sure the parameters such as op, renewer and service constant?
            106	  @Override public Token<DelegationTokenIdentifier> getDelegationToken(
            107	      String renewer) throws IOException {
            108	    URIBuilder uriBuilder =
            109	        new URIBuilder().setPath(DEFAULT_DELEGATION_TOKEN_MANAGER_ENDPOINT)
            110	            .addParameter("op", GET_DELEGATION_TOKEN_OP)
            111	            .addParameter("renewer", renewer)
            112	            .addParameter("service", WASB_DT_SERVICE_NAME.toString());
            113	    String responseBody = remoteCallHelper
            114	        .makeRemoteRequest(dtServiceUrls, uriBuilder.getPath(),
            115	            uriBuilder.getQueryParams(), HttpGet.METHOD_NAME);
            116	    return TokenUtils.toDelegationToken(JsonUtils.parse(responseBody));
            117	  }
            
          3. Do we need configurable retry policy? I'm not sure about this.
          4. In WasbRemoteCallHelper::shouldRetry(), I think we may need to self-interrupt and return in case of InterruptedException for the Thread.sleep(a.delayMillis). Currently we throw WasbRemoteCallException which may be too strict.
          5. This is a large patch. I'd appreciate if you can separate the code refactoring work for tokens from the retry logic. That will speed up the review and commit progress.

          Thanks,

          Show
          liuml07 Mingliang Liu added a comment - I will review this one shortly. Early minor comments: Can you make the @Override annotation in a separate line? The URI build code like following, can we make sure the parameters such as op , renewer and service constant? 106 @Override public Token<DelegationTokenIdentifier> getDelegationToken( 107 String renewer) throws IOException { 108 URIBuilder uriBuilder = 109 new URIBuilder().setPath(DEFAULT_DELEGATION_TOKEN_MANAGER_ENDPOINT) 110 .addParameter( "op" , GET_DELEGATION_TOKEN_OP) 111 .addParameter( "renewer" , renewer) 112 .addParameter( "service" , WASB_DT_SERVICE_NAME.toString()); 113 String responseBody = remoteCallHelper 114 .makeRemoteRequest(dtServiceUrls, uriBuilder.getPath(), 115 uriBuilder.getQueryParams(), HttpGet.METHOD_NAME); 116 return TokenUtils.toDelegationToken(JsonUtils.parse(responseBody)); 117 } Do we need configurable retry policy? I'm not sure about this. In WasbRemoteCallHelper::shouldRetry() , I think we may need to self-interrupt and return in case of InterruptedException for the Thread.sleep(a.delayMillis) . Currently we throw WasbRemoteCallException which may be too strict. This is a large patch. I'd appreciate if you can separate the code refactoring work for tokens from the retry logic. That will speed up the review and commit progress. Thanks,
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 14m 51s trunk passed
          +1 compile 0m 21s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 24s trunk passed
          +1 mvneclipse 0m 24s trunk passed
          +1 findbugs 0m 35s trunk passed
          +1 javadoc 0m 15s trunk passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 19s the patch passed
          -1 javac 0m 19s hadoop-tools_hadoop-azure generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)
          +1 checkstyle 0m 12s hadoop-tools/hadoop-azure: The patch generated 0 new + 28 unchanged - 23 fixed = 28 total (was 51)
          +1 mvnsite 0m 22s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 42s the patch passed
          -1 javadoc 0m 13s hadoop-azure in the patch failed.
          +1 unit 1m 29s hadoop-azure in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          22m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14443
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869844/HADOOP-14443.2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a73cfd81fda6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 1a56a3d
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt
          javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/testReport/
          modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 14m 51s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 24s trunk passed +1 mvneclipse 0m 24s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 15s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 19s the patch passed -1 javac 0m 19s hadoop-tools_hadoop-azure generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) +1 checkstyle 0m 12s hadoop-tools/hadoop-azure: The patch generated 0 new + 28 unchanged - 23 fixed = 28 total (was 51) +1 mvnsite 0m 22s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 42s the patch passed -1 javadoc 0m 13s hadoop-azure in the patch failed. +1 unit 1m 29s hadoop-azure in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 22m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14443 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869844/HADOOP-14443.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a73cfd81fda6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1a56a3d Default Java 1.8.0_131 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/artifact/patchprocess/diff-compile-javac-hadoop-tools_hadoop-azure.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/artifact/patchprocess/patch-javadoc-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12394/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          snayak Santhosh G Nayak added a comment -

          Thanks Steve Loughran for reviewing this. Addressed the code review comments.
          This patch is test against Azure West US endpoint.

          Show
          snayak Santhosh G Nayak added a comment - Thanks Steve Loughran for reviewing this. Addressed the code review comments. This patch is test against Azure West US endpoint.
          Hide
          stevel@apache.org Steve Loughran added a comment -
          1. I've not reviewed the actual functionality as that goes near UGI and the like. Someone who understands that will need to review it.
          2. We have the same policy for Azure patches as for the other object stores: can you state which endpoint you tested against (e.g . Azure ireland)
          3. hit the "submit patch" for Yetus to review

          General

          • the Line ending style checker is going to be unhappy. Please cut down where it doesn't destroy readability? Why: helps side-by-side review.
          • embrace Configuration.getTrimmedStrings()
          • SL4J construct strings automatically; critical for performance of debug log messages. Switch to LOG.debug("connecting to {}", uri) structure.

          RemoteSASKeyGeneratorImpl

          L124

          commaSeparatedUrlsString = conf.get(KEY_CRED_SERVICE_URLS);
          

          this should use conf.getTrimmedStrings() to have whitespace stripped, split done, tests for all this. Same for RemoteWasbAuthorizerImpl and RemoteWasbDelegationTokenManager

          L163: can you include the URI at fault in the exception text

          RemoteWasbAuthorizerImpl

          L157: Unless its always in inner messages, can you somehow include the URI/endpoint details in the wrapped exception. Your support team will appreciate this.
          L169: time to use <pre> in the javadocs.

          SecureWasbRemoteCallHelper

          L143. Why not make that commented out LOG.info an uncommented LOG.debug?

          L159. Use try-with-resources to manage closing of response

          L174/175. Log message at WARN, print the stack at DEBUG

          L217: SL4J construcst strings automatically; critical for performance of debug log messages. Switch to "connecting to {}", uri style.

          L225. Catching of any Exception is overbroad. Maybe: all IOEs pass up as is.

          JsonUtils

          L42. again, LOG.debug("JSON Parsing exception: {}", e.getMessage())
          L42. Maybe: log@debug the errant JSON string
          L43. String.toLowerCase needs to specify Locale.EN to work reliably round the world

          Show
          stevel@apache.org Steve Loughran added a comment - I've not reviewed the actual functionality as that goes near UGI and the like. Someone who understands that will need to review it. We have the same policy for Azure patches as for the other object stores: can you state which endpoint you tested against (e.g . Azure ireland) hit the "submit patch" for Yetus to review General the Line ending style checker is going to be unhappy. Please cut down where it doesn't destroy readability? Why: helps side-by-side review. embrace Configuration.getTrimmedStrings() SL4J construct strings automatically; critical for performance of debug log messages. Switch to LOG.debug("connecting to {}", uri) structure. RemoteSASKeyGeneratorImpl L124 commaSeparatedUrlsString = conf.get(KEY_CRED_SERVICE_URLS); this should use conf.getTrimmedStrings() to have whitespace stripped, split done, tests for all this. Same for RemoteWasbAuthorizerImpl and RemoteWasbDelegationTokenManager L163: can you include the URI at fault in the exception text RemoteWasbAuthorizerImpl L157: Unless its always in inner messages, can you somehow include the URI/endpoint details in the wrapped exception. Your support team will appreciate this. L169: time to use <pre> in the javadocs. SecureWasbRemoteCallHelper L143. Why not make that commented out LOG.info an uncommented LOG.debug? L159. Use try-with-resources to manage closing of response L174/175. Log message at WARN, print the stack at DEBUG L217: SL4J construcst strings automatically; critical for performance of debug log messages. Switch to "connecting to {}", uri style. L225. Catching of any Exception is overbroad. Maybe: all IOEs pass up as is. JsonUtils L42. again, LOG.debug("JSON Parsing exception: {}", e.getMessage()) L42. Maybe: log@debug the errant JSON string L43. String.toLowerCase needs to specify Locale.EN to work reliably round the world
          Hide
          snayak Santhosh G Nayak added a comment -

          Attaching initial version of the patch containing proposed changes.

          Show
          snayak Santhosh G Nayak added a comment - Attaching initial version of the patch containing proposed changes.

            People

            • Assignee:
              snayak Santhosh G Nayak
              Reporter:
              snayak Santhosh G Nayak
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development