Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.9.0, 3.0.0-beta1
-
None
Description
org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper.testWhenOneInstanceIsDown is failing for me on trunk
Attachments
Attachments
- HADOOP-14715-001.patch
- 0.6 kB
- Steve Loughran
- HADOOP-14715-002.patch
- 3 kB
- Esfandiar Manii
- HADOOP-14715-003.patch
- 2 kB
- Esfandiar Manii
Issue Links
- is broken by
-
HADOOP-14642 wasb: add support for caching Authorization and SASKeys
- Resolved
Activity
Cause of this is apparently secure mode being enabled by default. Commenting out that property in the XML file fixes things. Patch 001 does that, with a description on when to turn it on
Applying this patch to the parallelised test runs of HADOOP-14553 fixes it there with all the other tests completing; applying it to trunk fixes TestWasbRemoteCallHelper. Full test in progress; endpoint azure ireland
------------------------------------------------------- T E S T S ------------------------------------------------------- ------------------------------------------------------- T E S T S ------------------------------------------------------- Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 2.749 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 10
Thanks Steve, +1 from me. I will also follow up on TestWasbRemoteCallHelper and why it fails when fs.azure.secure.mode is true.
Full tests work. Notice how the test run is 4x slower than the parallel one
Results : Tests run: 773, Failures: 0, Errors: 0, Skipped: 155 [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 41:57 min (Wall Clock) [INFO] Finished at: 2017-08-03T19:02:32+01:00
Esfandiar, thanks for finding the regression! In addition to fixing the regression so that the test passes with fs.azure.secure.mode set to true or false, we should also change the test configuration back to how it was, so that fs.azure.secure.mode is false (which is what Steve did in HADOOP-14715-001.patch). The default should be false because that is the most common usage.
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 15m 15s | trunk passed |
Patch Compile Tests | |||
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
Other Tests | |||
+1 | unit | 0m 18s | hadoop-azure in the patch passed. |
+1 | asflicense | 0m 14s | The patch does not generate ASF License warnings. |
16m 28s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:14b5c93 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880267/HADOOP-14715-001.patch |
Optional Tests | asflicense unit xml |
uname | Linux 1fa53dd9f117 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f4c6b00 |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/testReport/ |
modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12946/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Updated the patch to handle scenarios where authorization caching is enabled/disabled and set secure mode to be disabled by default.
TestWasbRemoteCallHelper.java:
L62 - I recommend referencing CachingAuthorizer.KEY_AUTH_SERVICE_CACHING_ENABLE instead of declaring a new variable.
L362 - Duplication can be minimized, for example:
int expectedNumberOfInvocations = (isAuthorizationCachingEnabled) ? 1 : 2;
// then call times(expectedNumberOfInvocations)
looks reasonable. I've hit the submit button for yetus to review; I suspect it'll want lines cutting down.
As usual, what what was your test policy. Presumably you checked with both secure and insecure?
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 14s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 15m 0s | trunk passed |
+1 | compile | 0m 19s | trunk passed |
+1 | checkstyle | 0m 14s | trunk passed |
+1 | mvnsite | 0m 22s | trunk passed |
+1 | findbugs | 0m 35s | trunk passed |
+1 | javadoc | 0m 17s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 20s | the patch passed |
+1 | compile | 0m 21s | the patch passed |
+1 | javac | 0m 21s | the patch passed |
+1 | checkstyle | 0m 12s | the patch passed |
+1 | mvnsite | 0m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 0m 39s | the patch passed |
+1 | javadoc | 0m 13s | the patch passed |
Other Tests | |||
+1 | unit | 2m 8s | hadoop-azure in the patch passed. |
+1 | asflicense | 0m 13s | The patch does not generate ASF License warnings. |
22m 48s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:14b5c93 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880720/HADOOP-14715-003.patch |
Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml |
uname | Linux 018fa1e5c4dd 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 9891295 |
Default Java | 1.8.0_131 |
findbugs | v3.1.0-RC1 |
Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/testReport/ |
modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/12982/console |
Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
OK, checkstyle is happy. I just need the declaration of endpoint/options you've tested against so I can be confident it works
All the tests ran against:
wasb://testcontainer@xhdfs.blob.core.windows.net
When secure mode is on and authorization caching is enabled in azure-test.xml
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 137.797 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:20 min
[INFO] Finished at: 2017-08-08T18:18:36+00:00
[INFO] Final Memory: 22M/315M
[INFO] ------------------------------------------------------------------------
When secure mode is on and authorization caching is disabled in azure-test.xml
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 147.801 sec - in org.apache.hadoop.fs.azure.TestWasbRemoteCallHelper
Results :
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:32 min
[INFO] Finished at: 2017-08-08T18:24:54+00:00
[INFO] Final Memory: 35M/283M
[INFO] ------------------------------------------------------------------------
+1
committed to trunk; doing a built & test on branch-2 before committing there