Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs/azure
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fix to wasb:// (Azure) file system that allows the concurrent I/O feature to be used with the secure mode feature.
    • Flags:
      Patch

      Description

      This change allows the concurrent I/O feature (fs.azure.io.read.tolerate.concurrent.append = true) to work when secure mode is enabled (fs.azure.secure.mode = true).

      While running the test TestAzureConcurrentOutOfBandIo.testReadOOBWrites, I discovered that it fails when fs.azure.secure.mode = true with the error below:

      com.microsoft.azure.storage.StorageException: The condition specified using HTTP conditional header(s) is not met.
      at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:733)
      at com.microsoft.azure.storage.blob.BlobInputStream.dispatchRead(BlobInputStream.java:264)
      at com.microsoft.azure.storage.blob.BlobInputStream.readInternal(BlobInputStream.java:448)
      at com.microsoft.azure.storage.blob.BlobInputStream.read(BlobInputStream.java:420)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      at java.io.DataInputStream.read(DataInputStream.java:149)
      at org.apache.hadoop.fs.azure.TestAzureConcurrentOutOfBandIo.testReadOOBWrites(TestAzureConcurrentOutOfBandIo.java:167)

      There were a couple problems causing this failure:

      1) AzureNativeFileSystemStore.connectToAzureStorageInSecureMode was disabling concurrent I/O by setting fs.azure.io.read.tolerate.concurrent.append to false.

      2) SendRequestIntercept was unnecessarily updating the SAS for the request. Since this intercept only sets the request header "If-Match: *" to override the existing recondition, it is not necessary to update the SAS.

      The above issues have been fixed and a new test case has been added so that testReadOOBWrites now runs both with and without secure mode enabled.

      1. HADOOP-14546.003.patch
        13 kB
        Thomas Marquardt
      2. HADOOP-14546.002.patch
        14 kB
        Thomas Marquardt
      3. HADOOP-14546.001.patch
        14 kB
        Thomas Marquardt

        Activity

        Hide
        tmarquardt Thomas Marquardt added a comment -

        Attaching patch "HADOOP-14546.001.patch".

        Show
        tmarquardt Thomas Marquardt added a comment - Attaching patch " HADOOP-14546 .001.patch".
        Hide
        tmarquardt Thomas Marquardt added a comment -

        Submitting patch.

        Show
        tmarquardt Thomas Marquardt added a comment - Submitting patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 12m 44s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 19s trunk passed
        +1 findbugs 0m 25s trunk passed
        +1 javadoc 0m 11s trunk passed
        +1 mvninstall 0m 16s the patch passed
        +1 compile 0m 14s the patch passed
        +1 javac 0m 14s the patch passed
        -0 checkstyle 0m 9s hadoop-tools/hadoop-azure: The patch generated 14 new + 59 unchanged - 0 fixed = 73 total (was 59)
        +1 mvnsite 0m 16s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 31s the patch passed
        -1 javadoc 0m 11s hadoop-tools_hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 unit 1m 23s hadoop-azure in the patch passed.
        +1 asflicense 0m 14s The patch does not generate ASF License warnings.
        18m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14546
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873567/HADOOP-14546.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 156f17ef034c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3369540
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
        javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 12m 44s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 19s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed -0 checkstyle 0m 9s hadoop-tools/hadoop-azure: The patch generated 14 new + 59 unchanged - 0 fixed = 73 total (was 59) +1 mvnsite 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 31s the patch passed -1 javadoc 0m 11s hadoop-tools_hadoop-azure generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 unit 1m 23s hadoop-azure in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 18m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14546 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873567/HADOOP-14546.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 156f17ef034c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3369540 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt javadoc https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/artifact/patchprocess/diff-javadoc-javadoc-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12571/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        tmarquardt Thomas Marquardt added a comment -

        Attaching new patch (HADOOP-14546.002.patch) with fix for javadoc issue.

        Show
        tmarquardt Thomas Marquardt added a comment - Attaching new patch ( HADOOP-14546 .002.patch) with fix for javadoc issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 19m 32s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 12m 58s trunk passed
        +1 compile 0m 17s trunk passed
        +1 checkstyle 0m 12s trunk passed
        +1 mvnsite 0m 19s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 13s trunk passed
        +1 mvninstall 0m 18s the patch passed
        +1 compile 0m 17s the patch passed
        +1 javac 0m 17s the patch passed
        -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 14 new + 59 unchanged - 0 fixed = 73 total (was 59)
        +1 mvnsite 0m 19s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 36s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 1m 27s hadoop-azure in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        38m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14546
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873784/HADOOP-14546.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 66590ee04b35 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1a59847
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 19m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 12m 58s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 19s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 18s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 14 new + 59 unchanged - 0 fixed = 73 total (was 59) +1 mvnsite 0m 19s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 36s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 27s hadoop-azure in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 38m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14546 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12873784/HADOOP-14546.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 66590ee04b35 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1a59847 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12584/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        stevel@apache.org Steve Loughran added a comment -

        I see: you are essentially just removing the OOB feature in secure mode, and so stripping out bits of the codepath that are no longer needed. Looks OK.

        Some minor tuning in tests:

        AzureBlobStorageTestAccount L244: why use conf.set(KEY_USE_SECURE_MODE, String.valueOf(enableSecureMode)); over conf.setBoolean()?

        TestAzureConcurrentOutOfBandIoWithSecureMode : needless override of testReadOOBWrites()

        And I'll need you confirm you ran all the tests with secure set. Really the build needs a secure profile to test that, and the forthcoming testing doc to give more details on the test matrix.

        Show
        stevel@apache.org Steve Loughran added a comment - I see: you are essentially just removing the OOB feature in secure mode, and so stripping out bits of the codepath that are no longer needed. Looks OK. Some minor tuning in tests: AzureBlobStorageTestAccount L244: why use conf.set(KEY_USE_SECURE_MODE, String.valueOf(enableSecureMode)); over conf.setBoolean() ? TestAzureConcurrentOutOfBandIoWithSecureMode : needless override of testReadOOBWrites() And I'll need you confirm you ran all the tests with secure set. Really the build needs a secure profile to test that, and the forthcoming testing doc to give more details on the test matrix.
        Hide
        tmarquardt Thomas Marquardt added a comment -

        Thanks Steve, I attached HADOOP-14546.003.patch to address your feedback. I'll look into adding a secure profile as you mentioned. All /hadoop-tools/hadoop-azure tests pass with the patch:

        Tests run: 705, Failures: 0, Errors: 0, Skipped: 70

        However, I had to revert the hadoop-azure changes in commit 3369540653 (https://issues.apache.org/jira/browse/HADOOP-14296) as they broke the tests in TestFileSystemOperationsWithThreads.java and TestNativeAzureFileSystemClientLogging.java. I think this happened because the tests require an Azure Storage Account to run, and presumably that was not available when HADOOP-14296 was validated, so the tests were skipped. I will need to investigate this more to understand why those tests are broken, as well as a way to prevent this in the future.

        Show
        tmarquardt Thomas Marquardt added a comment - Thanks Steve, I attached HADOOP-14546 .003.patch to address your feedback. I'll look into adding a secure profile as you mentioned. All /hadoop-tools/hadoop-azure tests pass with the patch: Tests run: 705, Failures: 0, Errors: 0, Skipped: 70 However, I had to revert the hadoop-azure changes in commit 3369540653 ( https://issues.apache.org/jira/browse/HADOOP-14296 ) as they broke the tests in TestFileSystemOperationsWithThreads.java and TestNativeAzureFileSystemClientLogging.java. I think this happened because the tests require an Azure Storage Account to run, and presumably that was not available when HADOOP-14296 was validated, so the tests were skipped. I will need to investigate this more to understand why those tests are broken, as well as a way to prevent this in the future.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 12m 14s trunk passed
        +1 compile 0m 16s trunk passed
        +1 checkstyle 0m 11s trunk passed
        +1 mvnsite 0m 18s trunk passed
        +1 findbugs 0m 26s trunk passed
        +1 javadoc 0m 11s trunk passed
        +1 mvninstall 0m 17s the patch passed
        +1 compile 0m 16s the patch passed
        +1 javac 0m 16s the patch passed
        -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 15 new + 59 unchanged - 0 fixed = 74 total (was 59)
        +1 mvnsite 0m 18s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 34s the patch passed
        +1 javadoc 0m 10s the patch passed
        +1 unit 1m 26s hadoop-azure in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        18m 28s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HADOOP-14546
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874332/HADOOP-14546.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a3d12276ca68 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0111711
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/testReport/
        modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 12m 14s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 18s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 11s trunk passed +1 mvninstall 0m 17s the patch passed +1 compile 0m 16s the patch passed +1 javac 0m 16s the patch passed -0 checkstyle 0m 11s hadoop-tools/hadoop-azure: The patch generated 15 new + 59 unchanged - 0 fixed = 74 total (was 59) +1 mvnsite 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 1m 26s hadoop-azure in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 18m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14546 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12874332/HADOOP-14546.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a3d12276ca68 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0111711 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12619/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        +1

        I have committed to branch-2 and trunk branches. Thanks for your review Steve Loughran. Thanks for your contribution Thomas Marquardt.

        Show
        liuml07 Mingliang Liu added a comment - +1 I have committed to branch-2 and trunk branches. Thanks for your review Steve Loughran . Thanks for your contribution Thomas Marquardt .
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11937 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11937/)
        HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is (liuml07: rev 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)

        • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java
        • (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureConcurrentOutOfBandIo.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java
        • (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java
        • (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureConcurrentOutOfBandIoWithSecureMode.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11937 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11937/ ) HADOOP-14546 . Azure: Concurrent I/O does not work when secure.mode is (liuml07: rev 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28) (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/AzureBlobStorageTestAccount.java (edit) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureConcurrentOutOfBandIo.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/AzureNativeFileSystemStore.java (edit) hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azure/SendRequestIntercept.java (add) hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azure/TestAzureConcurrentOutOfBandIoWithSecureMode.java

          People

          • Assignee:
            tmarquardt Thomas Marquardt
            Reporter:
            tmarquardt Thomas Marquardt
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development