Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha3
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs/azure
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      The WASB FileSystem now uses version 5.3.0 of the Azure Storage SDK.
    • Flags:
      Patch

      Description

      Update WASB driver to use the latest version (5.3.0) of SDK for Microsoft Azure Storage Clients. We are currently using version 4.2.0 of the SDK.

      Azure Storage Clients changes between 4.2 and 5.3:

      • Fixed a bug where the transactional MD5 check would fail when downloading a range of blob or file and the recovery action is performed on a subsection of the range.
      • Fixed leaking connections for table requests.
      • Fixed a bug where retries happened immediately when experiencing a network exception uploading data or getting the response.
      • Fixed a bug where the response stream was not being closed on nonretryable exceptions.
      • Fixed Exists() calls on Shares and Directories to now populate metadata. This was already being done for Files.
      • Changed blob constants to support up to 256 MB on put blob for block blobs. The default value for put blob threshold has also been updated to half of the maximum, or 128 MB currently.
      • Fixed a bug that prevented setting content MD5 to true when creating a new file.
      • Fixed a bug where access conditions, options, and operation context were not being passed when calling openWriteExisting() on a page blob or a file.
      • Fixed a bug where an exception was being thrown on a range get of a blob or file when the options disableContentMD5Validation is set to false and useTransactionalContentMD5 is set to true and there is no overall MD5.
      • Fixed a bug where retries were happening immediately if a socket exception was thrown.
      • In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. This check has been moved to create() and uploadProperties() in CloudFileShare.
      • Prefix support for listing files and directories.
      • Added support for setting public access when creating a blob container
      • The public access setting on a blob container is now a container property returned from downloadProperties.
      • Add Message now modifies the PopReceipt, Id, NextVisibleTime, InsertionTime, and ExpirationTime properties of its CloudQueueMessage parameter.
      • Populate content MD5 for range gets on Blobs and Files.
      • Added support in Page Blob for incremental copy.
      • Added large BlockBlob upload support. Blocks can now support sizes up to 100 MB.
      • Added a new, memory-optimized upload strategy for the upload* APIs. This algorithm only applies for blocks greater than 4MB and when storeBlobContentMD5 and Client-Side Encryption are disabled.
      • getQualifiedUri() has been deprecated for Blobs. Please use getSnapshotQualifiedUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token.
      • getQualifiedStorageUri() has been deprecated for Blobs. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token.
      • Fixed a bug where copying from a blob that included a SAS token and a snapshot ommitted the SAS token.
      • Fixed a bug in client-side encryption for tables that was preventing the Java client from decrypting entities encrypted with the .NET client, and vice versa.
      • Added support for server-side encryption.
      • Added support for getBlobReferenceFromServer methods on CloudBlobContainer to support retrieving a blob without knowing its type.
      • Fixed a bug in the retry policies where 300 status codes were being retried when they shouldn't be.
      1. HADOOP-14536-002.patch
        0.4 kB
        Steve Loughran
      2. HADOOP-14536-01.patch
        0.5 kB
        Georgi Chalakov

        Issue Links

          Activity

          Hide
          Georgi Georgi Chalakov added a comment -

          -------------------------------------------------------
          T E S T S
          -------------------------------------------------------
          .
          .
          .
          Results :

          Tests run: 704, Failures: 0, Errors: 0, Skipped: 119

          Show
          Georgi Georgi Chalakov added a comment - ------------------------------------------------------- T E S T S ------------------------------------------------------- . . . Results : Tests run: 704, Failures: 0, Errors: 0, Skipped: 119
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 15m 43s trunk passed
          +1 compile 0m 11s trunk passed
          +1 mvnsite 0m 14s trunk passed
          +1 javadoc 0m 11s trunk passed
          +1 mvninstall 0m 8s the patch passed
          +1 compile 0m 7s the patch passed
          +1 javac 0m 7s the patch passed
          +1 mvnsite 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 javadoc 0m 8s the patch passed
          +1 unit 0m 7s hadoop-project in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          21m 29s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14536
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873566/HADOOP-14536-01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 90565c42f62f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3369540
          Default Java 1.8.0_131
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12574/testReport/
          modules C: hadoop-project U: hadoop-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12574/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 15m 43s trunk passed +1 compile 0m 11s trunk passed +1 mvnsite 0m 14s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 8s the patch passed +1 compile 0m 7s the patch passed +1 javac 0m 7s the patch passed +1 mvnsite 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 javadoc 0m 8s the patch passed +1 unit 0m 7s hadoop-project in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 21m 29s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14536 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873566/HADOOP-14536-01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 90565c42f62f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3369540 Default Java 1.8.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12574/testReport/ modules C: hadoop-project U: hadoop-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12574/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment - - edited

          Now I'm confused. Why not just submit this patch for HADOOP-14490 (we'll just change its title to the exact JAR thats in)?

          Also, need declaration of the specific azure endpoint you ran all the tests against

          Show
          stevel@apache.org Steve Loughran added a comment - - edited Now I'm confused. Why not just submit this patch for HADOOP-14490 (we'll just change its title to the exact JAR thats in)? Also, need declaration of the specific azure endpoint you ran all the tests against
          Hide
          Georgi Georgi Chalakov added a comment -

          The patch changes the version of Azure Client SDK. The new SDK offers fixes and new functionality, but it should not change current behavior, so it doesn't require modification of existing tests or new tests. All tests pass when the change was tested against wasb://testhdfs@hdfs4.blob.core.windows.net

          Show
          Georgi Georgi Chalakov added a comment - The patch changes the version of Azure Client SDK. The new SDK offers fixes and new functionality, but it should not change current behavior, so it doesn't require modification of existing tests or new tests. All tests pass when the change was tested against wasb://testhdfs@hdfs4.blob.core.windows.net
          Hide
          stevel@apache.org Steve Loughran added a comment -

          OK. Patch isn't applying to trunk, oddly enough

          git apx ~/hadoop-patches/incoming/HADOOP-14536-01.patch 
          /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:9: trailing whitespace.
                <dependency>
          /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:10: trailing whitespace.
                  <groupId>com.microsoft.azure</groupId>
          /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:11: trailing whitespace.
                  <artifactId>azure-storage</artifactId>
          /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:13: trailing whitespace.
                  <version>5.3.0</version>
          /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:14: trailing whitespace.
               </dependency>
          Checking patch pom.xml...
          error: while searching for:
                <dependency>
                  <groupId>com.microsoft.azure</groupId>
                  <artifactId>azure-storage</artifactId>
                  <version>4.2.0</version>
               </dependency>
          
                <dependency>
          
          error: patch failed: pom.xml:1114
          error: repository lacks the necessary blob to fall back on 3-way merge.
          error: pom.xml: patch does not apply
          
          Show
          stevel@apache.org Steve Loughran added a comment - OK. Patch isn't applying to trunk, oddly enough git apx ~/hadoop-patches/incoming/HADOOP-14536-01.patch /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:9: trailing whitespace. <dependency> /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:10: trailing whitespace. <groupId>com.microsoft.azure</groupId> /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:11: trailing whitespace. <artifactId>azure-storage</artifactId> /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:13: trailing whitespace. <version>5.3.0</version> /Users/stevel/hadoop-patches/incoming/HADOOP-14536-01.patch:14: trailing whitespace. </dependency> Checking patch pom.xml... error: while searching for : <dependency> <groupId>com.microsoft.azure</groupId> <artifactId>azure-storage</artifactId> <version>4.2.0</version> </dependency> <dependency> error: patch failed: pom.xml:1114 error: repository lacks the necessary blob to fall back on 3-way merge. error: pom.xml: patch does not apply
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Preparing new patch. Test run works except for those tests failing under HADOOP-14573

          Results :
          
          Failed tests: 
            TestFileSystemOperationsWithThreads.testDeleteLargeFolderDisableThreads:307 null
            TestFileSystemOperationsWithThreads.testRenameThreadPoolExceptionFailure:559 null
            TestFileSystemOperationsWithThreads.testDeleteSmallFolderDisableThreadsDisableFlatListing:332 null
            TestFileSystemOperationsWithThreads.testRenameLargeFolderDisableThreads:187 null
            TestFileSystemOperationsWithThreads.testRenameThreadPoolExecuteFailure:590 null
            TestFileSystemOperationsWithThreads.testDeleteThreadPoolExecuteFailure:396 null
            TestFileSystemOperationsWithThreads.testDeleteLargeFolderWithThreads:283 null
            TestFileSystemOperationsWithThreads.testRenameSingleRenameException:710 null
            TestFileSystemOperationsWithThreads.testDeleteSingleDeleteFailure:499 null
            TestFileSystemOperationsWithThreads.testDeleteThreadPoolExecuteSingleThreadFailure:430 null
            TestFileSystemOperationsWithThreads.testDeleteThreadPoolTerminationFailure:475 null
            TestFileSystemOperationsWithThreads.testDeleteSingleDeleteException:531 null
            TestFileSystemOperationsWithThreads.testRenameLargeFolderWithThreads:161 null
            TestFileSystemOperationsWithThreads.testRenameThreadPoolTerminationFailure:669 null
            TestFileSystemOperationsWithThreads.testDeleteSmallFolderWithThreads:257 null
            TestFileSystemOperationsWithThreads.testRenameThreadPoolExecuteSingleThreadFailure:624 null
            TestFileSystemOperationsWithThreads.testRenameSmallFolderWithThreads:134 null
            TestFileSystemOperationsWithThreads.testDeleteThreadPoolExceptionFailure:364 null
            TestFileSystemOperationsWithThreads.testRenameSmallFolderDisableThreadsDisableFlatListing:213 null
            TestNativeAzureFileSystemClientLogging.testLoggingEnabled:108 null
          
          Tests run: 698, Failures: 20, Errors: 0, Skipped: 119
          
          Show
          stevel@apache.org Steve Loughran added a comment - Preparing new patch. Test run works except for those tests failing under HADOOP-14573 Results : Failed tests: TestFileSystemOperationsWithThreads.testDeleteLargeFolderDisableThreads:307 null TestFileSystemOperationsWithThreads.testRenameThreadPoolExceptionFailure:559 null TestFileSystemOperationsWithThreads.testDeleteSmallFolderDisableThreadsDisableFlatListing:332 null TestFileSystemOperationsWithThreads.testRenameLargeFolderDisableThreads:187 null TestFileSystemOperationsWithThreads.testRenameThreadPoolExecuteFailure:590 null TestFileSystemOperationsWithThreads.testDeleteThreadPoolExecuteFailure:396 null TestFileSystemOperationsWithThreads.testDeleteLargeFolderWithThreads:283 null TestFileSystemOperationsWithThreads.testRenameSingleRenameException:710 null TestFileSystemOperationsWithThreads.testDeleteSingleDeleteFailure:499 null TestFileSystemOperationsWithThreads.testDeleteThreadPoolExecuteSingleThreadFailure:430 null TestFileSystemOperationsWithThreads.testDeleteThreadPoolTerminationFailure:475 null TestFileSystemOperationsWithThreads.testDeleteSingleDeleteException:531 null TestFileSystemOperationsWithThreads.testRenameLargeFolderWithThreads:161 null TestFileSystemOperationsWithThreads.testRenameThreadPoolTerminationFailure:669 null TestFileSystemOperationsWithThreads.testDeleteSmallFolderWithThreads:257 null TestFileSystemOperationsWithThreads.testRenameThreadPoolExecuteSingleThreadFailure:624 null TestFileSystemOperationsWithThreads.testRenameSmallFolderWithThreads:134 null TestFileSystemOperationsWithThreads.testDeleteThreadPoolExceptionFailure:364 null TestFileSystemOperationsWithThreads.testRenameSmallFolderDisableThreadsDisableFlatListing:213 null TestNativeAzureFileSystemClientLogging.testLoggingEnabled:108 null Tests run: 698, Failures: 20, Errors: 0, Skipped: 119
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 002; this is just 001 applied via the IDE and now applying to trunk via git apply

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 002; this is just 001 applied via the IDE and now applying to trunk via git apply
          Hide
          stevel@apache.org Steve Loughran added a comment -

          ps: time to execute tests is 2h on a remote machine with 60 mbs/down, 18 up. it should be able to complete in less than ten minutes.

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 01:50 h (Wall Clock)
          [INFO] Finished at: 2017-06-27T13:58:45+01:00
          [INFO] Final Memory: 23M/785M
          [INFO] ------------------------------------------------------------------------
          
          Show
          stevel@apache.org Steve Loughran added a comment - ps: time to execute tests is 2h on a remote machine with 60 mbs/down, 18 up. it should be able to complete in less than ten minutes. [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:50 h (Wall Clock) [INFO] Finished at: 2017-06-27T13:58:45+01:00 [INFO] Final Memory: 23M/785M [INFO] ------------------------------------------------------------------------
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 13m 24s trunk passed
          +1 compile 0m 10s trunk passed
          +1 mvnsite 0m 11s trunk passed
          +1 javadoc 0m 11s trunk passed
          +1 mvninstall 0m 8s the patch passed
          +1 compile 0m 6s the patch passed
          +1 javac 0m 6s the patch passed
          +1 mvnsite 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 javadoc 0m 7s the patch passed
          +1 unit 0m 7s hadoop-project in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          15m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14536
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874682/HADOOP-14536-002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
          uname Linux 22b2336bca1e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07defa4
          Default Java 1.8.0_131
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12639/testReport/
          modules C: hadoop-project U: hadoop-project
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12639/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 24s trunk passed +1 compile 0m 10s trunk passed +1 mvnsite 0m 11s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 8s the patch passed +1 compile 0m 6s the patch passed +1 javac 0m 6s the patch passed +1 mvnsite 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 7s the patch passed +1 unit 0m 7s hadoop-project in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 15m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14536 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874682/HADOOP-14536-002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 22b2336bca1e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07defa4 Default Java 1.8.0_131 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12639/testReport/ modules C: hadoop-project U: hadoop-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12639/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11932 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11932/)
          HADOOP-14536. Update azure-storage sdk to version 5.3.0 Contributed by (stevel: rev 94e39c6c4e39efadb72d7765ad3cc4ba43e0687a)

          • (edit) hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11932 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11932/ ) HADOOP-14536 . Update azure-storage sdk to version 5.3.0 Contributed by (stevel: rev 94e39c6c4e39efadb72d7765ad3cc4ba43e0687a) (edit) hadoop-project/pom.xml
          Hide
          liuml07 Mingliang Liu added a comment -

          Thanks Steve for review and commit. As usual, I link HADOOP-9991 a dependent of this.

          Show
          liuml07 Mingliang Liu added a comment - Thanks Steve for review and commit. As usual, I link HADOOP-9991 a dependent of this.

            People

            • Assignee:
              Georgi Georgi Chalakov
              Reporter:
              liuml07 Mingliang Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development