Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.9.0, 3.0.0-beta1
    • 2.9.0, 3.0.0-beta1
    • fs/azure, test
    • None

    Description

      org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown is failing for me on trunk

      Attachments

        1. HADOOP-14715-001.patch
          0.6 kB
          Steve Loughran
        2. HADOOP-14715-002.patch
          3 kB
          Esfandiar Manii
        3. HADOOP-14715-003.patch
          2 kB
          Esfandiar Manii

        Issue Links

          Activity

            stevel@apache.org Steve Loughran added a comment -
            Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 165.074 sec <<< FAILURE! - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            testWhenOneInstanceIsDown(org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper)  Time elapsed: 3.469 sec  <<< FAILURE!
            org.mockito.exceptions.verification.TooLittleActualInvocations: 
            httpClient.execute(
                <Http get for service local>
            );
            Wanted 2 times:
            -> at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359)
            But was 1 time:
            -> at org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(WasbRemoteCallHelper.java:151)
            
            	at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359)
            
            
            Results :
            
            Failed tests: 
              TestWasbRemoteCallHelper.testWhenOneInstanceIsDown:359 
            httpClient.execute(
                <Http get for service local>
            );
            Wanted 2 times:
            -> at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359)
            But was 1 time:
            -> at org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(WasbRemoteCallHelper.java:151)
            
            stevel@apache.org Steve Loughran added a comment - Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 165.074 sec <<< FAILURE! - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper testWhenOneInstanceIsDown(org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper) Time elapsed: 3.469 sec <<< FAILURE! org.mockito.exceptions.verification.TooLittleActualInvocations: httpClient.execute( <Http get for service local> ); Wanted 2 times: -> at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359) But was 1 time: -> at org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(WasbRemoteCallHelper.java:151) at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359) Results : Failed tests: TestWasbRemoteCallHelper.testWhenOneInstanceIsDown:359 httpClient.execute( <Http get for service local> ); Wanted 2 times: -> at org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown(TestWasbRemoteCallHelper.java:359) But was 1 time: -> at org.apache.hadoop.fs.azure.WasbRemoteCallHelper.retryableRequest(WasbRemoteCallHelper.java:151)
            stevel@apache.org Steve Loughran added a comment -

            cause is probably HADOOP-14642

            stevel@apache.org Steve Loughran added a comment - cause is probably HADOOP-14642
            stevel@apache.org Steve Loughran added a comment -

            Cause of this is apparently secure mode being enabled by default. Commenting out that property in the XML file fixes things. Patch 001 does that, with a description on when to turn it on

            Applying this patch to the parallelised test runs of HADOOP-14553 fixes it there with all the other tests completing; applying it to trunk fixes TestWasbRemoteCallHelper. Full test in progress; endpoint azure ireland

            -------------------------------------------------------
             T E S T S
            -------------------------------------------------------
            
            -------------------------------------------------------
             T E S T S
            -------------------------------------------------------
            Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.749 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            
            Results :
            
            Tests run: 10, Failures: 0, Errors: 0, Skipped: 10
            
            stevel@apache.org Steve Loughran added a comment - Cause of this is apparently secure mode being enabled by default. Commenting out that property in the XML file fixes things. Patch 001 does that, with a description on when to turn it on Applying this patch to the parallelised test runs of HADOOP-14553 fixes it there with all the other tests completing; applying it to trunk fixes TestWasbRemoteCallHelper . Full test in progress; endpoint azure ireland ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.749 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 10

            Thanks Steve, +1 from me. I will also follow up on TestWasbRemoteCallHelper and why it fails when fs.azure.secure.mode is true.

            tmarquardt Thomas Marqardt added a comment - Thanks Steve, +1 from me. I will also follow up on TestWasbRemoteCallHelper and why it fails when fs.azure.secure.mode is true.
            stevel@apache.org Steve Loughran added a comment -

            Full tests work. Notice how the test run is 4x slower than the parallel one

            Results :
            
            Tests run: 773, Failures: 0, Errors: 0, Skipped: 155
            
            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 41:57 min (Wall Clock)
            [INFO] Finished at: 2017-08-03T19:02:32+01:00
            
            stevel@apache.org Steve Loughran added a comment - Full tests work. Notice how the test run is 4x slower than the parallel one Results : Tests run: 773, Failures: 0, Errors: 0, Skipped: 155 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 41:57 min (Wall Clock) [INFO] Finished at: 2017-08-03T19:02:32+01:00

            This is a regression, I am preparing a fix

            esmanii Esfandiar Manii added a comment - This is a regression, I am preparing a fix

            Esfandiar, thanks for finding the regression! In addition to fixing the regression so that the test passes with fs.azure.secure.mode set to true or false, we should also change the test configuration back to how it was, so that fs.azure.secure.mode is false (which is what Steve did in HADOOP-14715-001.patch). The default should be false because that is the most common usage.

            tmarquardt Thomas Marqardt added a comment - Esfandiar, thanks for finding the regression! In addition to fixing the regression so that the test passes with fs.azure.secure.mode set to true or false , we should also change the test configuration back to how it was, so that fs.azure.secure.mode is false (which is what Steve did in HADOOP-14715 -001.patch). The default should be false because that is the most common usage.
            hadoopqa Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 15m 15s trunk passed
                  Patch Compile Tests
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
                  Other Tests
            +1 unit 0m 18s hadoop-azure in the patch passed.
            +1 asflicense 0m 14s The patch does not generate ASF License warnings.
            16m 28s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue HADOOP-14715
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880267/HADOOP-14715-001.patch
            Optional Tests asflicense unit xml
            uname Linux 1fa53dd9f117 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / f4c6b00
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/testReport/
            modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 15s trunk passed       Patch Compile Tests +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file.       Other Tests +1 unit 0m 18s hadoop-azure in the patch passed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 16m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14715 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880267/HADOOP-14715-001.patch Optional Tests asflicense unit xml uname Linux 1fa53dd9f117 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f4c6b00 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            stevel@apache.org Steve Loughran added a comment -

            esmanii assigned to you; I'll review the patch when it's ready

            stevel@apache.org Steve Loughran added a comment - esmanii assigned to you; I'll review the patch when it's ready

            Updated the patch to handle scenarios where authorization caching is enabled/disabled and set secure mode to be disabled by default.

            esmanii Esfandiar Manii added a comment - Updated the patch to handle scenarios where authorization caching is enabled/disabled and set secure mode to be disabled by default.

            TestWasbRemoteCallHelper.java:
            L62 - I recommend referencing CachingAuthorizer.KEY_AUTH_SERVICE_CACHING_ENABLE instead of declaring a new variable.

            L362 - Duplication can be minimized, for example:

            int expectedNumberOfInvocations = (isAuthorizationCachingEnabled) ? 1 : 2;

            // then call times(expectedNumberOfInvocations)

            tmarquardt Thomas Marqardt added a comment - TestWasbRemoteCallHelper.java : L62 - I recommend referencing CachingAuthorizer.KEY_AUTH_SERVICE_CACHING_ENABLE instead of declaring a new variable. L362 - Duplication can be minimized, for example: int expectedNumberOfInvocations = (isAuthorizationCachingEnabled) ? 1 : 2; // then call times(expectedNumberOfInvocations)
            esmanii Esfandiar Manii added a comment - - edited

            Updated wrt comments from Thomas

            esmanii Esfandiar Manii added a comment - - edited Updated wrt comments from Thomas

            Looks good. +1

            tmarquardt Thomas Marqardt added a comment - Looks good. +1
            stevel@apache.org Steve Loughran added a comment -

            looks reasonable. I've hit the submit button for yetus to review; I suspect it'll want lines cutting down.

            As usual, what what was your test policy. Presumably you checked with both secure and insecure?

            stevel@apache.org Steve Loughran added a comment - looks reasonable. I've hit the submit button for yetus to review; I suspect it'll want lines cutting down. As usual, what what was your test policy. Presumably you checked with both secure and insecure?
            hadoopqa Hadoop QA added a comment -
            +1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 14s Docker mode activated.
                  Prechecks
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
                  trunk Compile Tests
            +1 mvninstall 15m 0s trunk passed
            +1 compile 0m 19s trunk passed
            +1 checkstyle 0m 14s trunk passed
            +1 mvnsite 0m 22s trunk passed
            +1 findbugs 0m 35s trunk passed
            +1 javadoc 0m 17s trunk passed
                  Patch Compile Tests
            +1 mvninstall 0m 20s the patch passed
            +1 compile 0m 21s the patch passed
            +1 javac 0m 21s the patch passed
            +1 checkstyle 0m 12s the patch passed
            +1 mvnsite 0m 22s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 0m 39s the patch passed
            +1 javadoc 0m 13s the patch passed
                  Other Tests
            +1 unit 2m 8s hadoop-azure in the patch passed.
            +1 asflicense 0m 13s The patch does not generate ASF License warnings.
            22m 48s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:14b5c93
            JIRA Issue HADOOP-14715
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880720/HADOOP-14715-003.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
            uname Linux 018fa1e5c4dd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 9891295
            Default Java 1.8.0_131
            findbugs v3.1.0-RC1
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/testReport/
            modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/console
            Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 15m 0s trunk passed +1 compile 0m 19s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 findbugs 0m 35s trunk passed +1 javadoc 0m 17s trunk passed       Patch Compile Tests +1 mvninstall 0m 20s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 0m 39s the patch passed +1 javadoc 0m 13s the patch passed       Other Tests +1 unit 2m 8s hadoop-azure in the patch passed. +1 asflicense 0m 13s The patch does not generate ASF License warnings. 22m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14715 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880720/HADOOP-14715-003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 018fa1e5c4dd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9891295 Default Java 1.8.0_131 findbugs v3.1.0-RC1 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/testReport/ modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            stevel@apache.org Steve Loughran added a comment -

            OK, checkstyle is happy. I just need the declaration of endpoint/options you've tested against so I can be confident it works

            stevel@apache.org Steve Loughran added a comment - OK, checkstyle is happy. I just need the declaration of endpoint/options you've tested against so I can be confident it works

            All the tests ran against:
            wasb://testcontainer@xhdfs.blob.core.windows.net

            When secure mode is on and authorization caching is enabled in azure-test.xml
            -------------------------------------------------------
            T E S T S
            -------------------------------------------------------

            -------------------------------------------------------
            T E S T S
            -------------------------------------------------------
            Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 137.797 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper

            Results :

            Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 02:20 min
            [INFO] Finished at: 2017-08-08T18:18:36+00:00
            [INFO] Final Memory: 22M/315M
            [INFO] ------------------------------------------------------------------------

            When secure mode is on and authorization caching is disabled in azure-test.xml
            -------------------------------------------------------
            T E S T S
            -------------------------------------------------------

            -------------------------------------------------------
            T E S T S
            -------------------------------------------------------
            Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
            Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.801 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper

            Results :

            Tests run: 10, Failures: 0, Errors: 0, Skipped: 0

            [INFO] ------------------------------------------------------------------------
            [INFO] BUILD SUCCESS
            [INFO] ------------------------------------------------------------------------
            [INFO] Total time: 02:32 min
            [INFO] Finished at: 2017-08-08T18:24:54+00:00
            [INFO] Final Memory: 35M/283M
            [INFO] ------------------------------------------------------------------------

            esmanii Esfandiar Manii added a comment - All the tests ran against: wasb://testcontainer@xhdfs.blob.core.windows.net When secure mode is on and authorization caching is enabled in azure-test.xml ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 137.797 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:20 min [INFO] Finished at: 2017-08-08T18:18:36+00:00 [INFO] Final Memory: 22M/315M [INFO] ------------------------------------------------------------------------ When secure mode is on and authorization caching is disabled in azure-test.xml ------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.801 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:32 min [INFO] Finished at: 2017-08-08T18:24:54+00:00 [INFO] Final Memory: 35M/283M [INFO] ------------------------------------------------------------------------
            stevel@apache.org Steve Loughran added a comment -

            +1

            committed to trunk; doing a built & test on branch-2 before committing there

            stevel@apache.org Steve Loughran added a comment - +1 committed to trunk; doing a built & test on branch-2 before committing there
            stevel@apache.org Steve Loughran added a comment -

            committed to branch-2 too. Thanks!

            stevel@apache.org Steve Loughran added a comment - committed to branch-2 too. Thanks!

            People

              esmanii Esfandiar Manii
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: