Details
Description
YARN-3893 adds this test, that relies on some CapacityScheduler-specific stuff for refreshAll to fail, which doesn't apply when using FairScheduler.
Issue Links
- is related to
-
YARN-5333
Some recovered apps are put into default queue when RM HA
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
IIUC, the test is using CS to force a failure of refreshAll. If that is indeed the case, we could just override the MockRM to use an AdminService that just fails the refreshAll? By the way, I haven't started on it yet, so please feel free to take it up.
Apologies for not considering case whr default scheduler could be FairScheduler . Attaching patch to handle the same .
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 13s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 40s | trunk passed |
| +1 | compile | 0m 28s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 0m 30s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 17s | trunk passed |
| +1 | mvnsite | 0m 35s | trunk passed |
| +1 | mvneclipse | 0m 14s | trunk passed |
| +1 | findbugs | 1m 4s | trunk passed |
| +1 | javadoc | 0m 22s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 25s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 30s | the patch passed |
| +1 | compile | 0m 26s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 0m 26s | the patch passed |
| +1 | compile | 0m 26s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 0m 26s | the patch passed |
| +1 | checkstyle | 0m 16s | the patch passed |
| +1 | mvnsite | 0m 32s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 15s | the patch passed |
| +1 | javadoc | 0m 20s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 24s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 79m 59s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 58m 49s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 18s | Patch does not generate ASF License warnings. |
| 155m 19s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| hadoop.yarn.webapp.TestRMWithCSRFFilter | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
| JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens | |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| hadoop.yarn.webapp.TestRMWithCSRFFilter | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerPreemption | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler |
This message was automatically generated.
Thanks for picking this up, Bibin A Chundatt.
Few comments:
- AdminService: since the test is in the same class, refreshAll could be package-private instead of public. Also, you might want to mark it @VisibleForTesting along with a comment that it can be private otherwise.
- TestRMHA
- The new variable counter could be private to the anonymous AdminService class we are creating in the test.
- The assertion when the RM fails to transition to active seems backwards. Shouldn't we be checking e.getMessage().contains("<blah>")?
- I wonder if we are even running into that exception. If the test is expecting the exception, we should add an Assert.fail right after the call to transition to active.
- Also, I am not a fan of checking just the message verbatim. Can we check if the exception is ServiceFailedException and preferably the expected RM state (Active/Standby)?
- Not introduced in this patch, but the asserts in the test should have a corresponding error message to explain what exactly is going on .
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 48s | trunk passed |
| +1 | compile | 0m 29s | trunk passed with JDK v1.8.0_77 |
| +1 | compile | 0m 29s | trunk passed with JDK v1.7.0_95 |
| +1 | checkstyle | 0m 18s | trunk passed |
| +1 | mvnsite | 0m 34s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 4s | trunk passed |
| +1 | javadoc | 0m 22s | trunk passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 26s | trunk passed with JDK v1.7.0_95 |
| +1 | mvninstall | 0m 29s | the patch passed |
| +1 | compile | 0m 24s | the patch passed with JDK v1.8.0_77 |
| +1 | javac | 0m 24s | the patch passed |
| +1 | compile | 0m 26s | the patch passed with JDK v1.7.0_95 |
| +1 | javac | 0m 26s | the patch passed |
| +1 | checkstyle | 0m 16s | the patch passed |
| +1 | mvnsite | 0m 33s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 1m 14s | the patch passed |
| +1 | javadoc | 0m 19s | the patch passed with JDK v1.8.0_77 |
| +1 | javadoc | 0m 24s | the patch passed with JDK v1.7.0_95 |
| -1 | unit | 76m 28s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_77. |
| -1 | unit | 55m 4s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_95. |
| +1 | asflicense | 0m 19s | Patch does not generate ASF License warnings. |
| 148m 10s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_77 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| hadoop.yarn.webapp.TestRMWithCSRFFilter | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
| JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes |
| JDK v1.7.0_95 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesForCSWithPartitions | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens | |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| hadoop.yarn.webapp.TestRMWithCSRFFilter | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
| hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps | |
| hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler |
This message was automatically generated.
Testcase failures are not related to patch attached
+1
FAILURE: Integrated in Hadoop-trunk-Commit #9587 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9587/)
YARN-4927. TestRMHA#testTransitionedToActiveRefreshFail fails with (kasha: rev ff95fd547b3030529983fa4e701ac728c54ab63a)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHA.java
Thanks Bibin for fixing this up. Just committed to trunk and branch-2.
Backporting to branch-2.8 and branch-2.7 as it fixes some test errors there as well.
Thanks Karthik for finding out test failure. Even in HadoopQA result could not find because of test runs using default configurations. I think there are 2 options to handle it, firstly test class can run with parameterized. But only 1 test case depends on CS and rest all other tests are independent of any scheduler specific. Secondly, this particular test case can be moved out OR skip if fair scheduler is used.