Details
Description
This is the 2nd dependent jira of YARN-8059.
As updating the metrics is an independent step from handling preemption, this jira only deals with the queue metrics update of custom resources.
The following metrics should be updated:
- allocated resources
- available resources
- pending resources
- reserved resources
- aggregate seconds preempted
Attachments
Attachments
- YARN-8842.001.patch
- 74 kB
- Szilard Nemeth
- YARN-8842.002.patch
- 46 kB
- Szilard Nemeth
- YARN-8842.003.patch
- 82 kB
- Szilard Nemeth
- YARN-8842.004.patch
- 82 kB
- Szilard Nemeth
- YARN-8842.005.patch
- 82 kB
- Szilard Nemeth
- YARN-8842.006.patch
- 108 kB
- Szilard Nemeth
- YARN-8842.007.patch
- 99 kB
- Szilard Nemeth
- YARN-8842.008.patch
- 99 kB
- Szilard Nemeth
- YARN-8842.009.patch
- 97 kB
- Szilard Nemeth
- YARN-8842.010.patch
- 93 kB
- Szilard Nemeth
- YARN-8842.011.patch
- 96 kB
- Szilard Nemeth
- YARN-8842.012.patch
- 96 kB
- Szilard Nemeth
- YARN-8842-branch-2.001.patch
- 102 kB
- Jonathan Hung
- YARN-8842-branch-2.002.patch
- 102 kB
- Jonathan Hung
- YARN-8842-branch-2.003.patch
- 102 kB
- Jonathan Hung
Issue Links
Activity
Thank you for splitting this of from the pre-emption jira.
I tried to apply the patch on top of YARN-8750 but it did not work...
Beside that I went through the code as a diff and already have a couple of remarks:
- You are always creating a new QueueMetricsForCustomResources. In the case that we just have two resources (mem and vcores) we will never use it and all metrics will stay zero. We do have the full overhead of making all the calls. From a performance perspective would it not be better to determine if we have custom resources configured in the constructor and set a flag to update the custom metrics or not? We can then make updating the custom resource metrics conditional which would keep the performance impact in most clusters to a minimum.
- In updatePreemptedSecondsForCustomResources() you directly set the parent metrics but you do not take into account that there might be multiple levels above the queue you are working on. We need to call parent.updatePreemptedSecondsForCustomResources()
- In getAllocatedResources() you're using BuilderUtils if you have no custom resources and a new resource object directly if you have. We should either use BuilderUtils in both cases or neither for consistency.
- Again getAllocatedResources() when we have custom resource types (even if none are allocated) I would expect a metrics object back showing all the resource types not just memory and vcores. There should also be no need for isThereAnyAllocatedResource the Resource handles that itself.
- In the test cases I could not find any multiple queue levels tests. root -> parent -> leaf. At least we should check that case with the root queue metrics to catch the issue in the updatePreemptedSecondsForCustomResources case from above.
I'll dig further into the tests when we have a patch that applies.
One further remark that showed up while reviewing YARN-8059: I don't think we should have the getMaxAllocationUtilization in the custom metrics and getMaxUtilizationOfCustomResources in the normal metrics. They pollute the classes with calculations that we should keep in the FairScheduler only.
Hi wilfreds!
Thanks for the comments.
1. Good idea to decide on the flag. However, it's not immediately straightforward for me whether we only need to treat memory / cores as default resources. See: https://github.com/apache/hadoop/blob/12a095a496dd59066d73a7a6c24129b5b6a9d650/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceInformation.java#L56 , here we also have GPU / FPGA in the mandatory resources list.
How would you exactly check in the constructor whether we have custom resources configured?
I can think of the following: As we receive the configuration object in the constructor, we could check for how many keys are in there for YarnConfiguration.RESOURCE_TYPES. If it's 2 (4 in case we treat GPU / FPGA as standard resources), then we are not creating the QueueMetricsForCustomResources, otherwise we create it.
Can you think of any better way to do this?
2. It's a good catch, thanks!
3. Good point, fixed.
4. Refactored QueueMetricsTestcase in order to handle more than 2 queues, so that I can not only specify root and leaf queues, but multiple queues in a hierarchy.
About your second comment: I think it does make sense to move that method with YARN-8059.
Thanks snemeth for the patch.
For a clean-code perspective, QueueMetricsForCustomResources can include memory and cpu metrics too. Though I am not sure how that's going to impact the compatibility (not sure if @Metric annotation is some sort of API) and performance that wilfreds is concerned with.
We can also split QueueMetricsForCustomResources into three instances, Pending, Available and Allocated, so it is more readable.
I'll take another pass to look at the uni tests added later.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 27s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 17m 37s | trunk passed |
+1 | compile | 0m 42s | trunk passed |
+1 | checkstyle | 0m 37s | trunk passed |
+1 | mvnsite | 0m 47s | trunk passed |
+1 | shadedclient | 12m 4s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 10s | trunk passed |
+1 | javadoc | 0m 29s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 44s | the patch passed |
+1 | compile | 0m 38s | the patch passed |
+1 | javac | 0m 38s | the patch passed |
-0 | checkstyle | 0m 33s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 42 new + 142 unchanged - 2 fixed = 184 total (was 144) |
+1 | mvnsite | 0m 43s | the patch passed |
-1 | whitespace | 0m 0s | The patch has 22 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
+1 | shadedclient | 11m 38s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 18s | the patch passed |
+1 | javadoc | 0m 27s | the patch passed |
Other Tests | |||
-1 | unit | 89m 27s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
139m 36s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12942640/YARN-8842.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 999dfc22c67c 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / cdf5d58 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22076/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/22076/artifact/out/whitespace-eol.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22076/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22076/testReport/ |
Max. process+thread count | 879 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22076/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | patch | 0m 6s | |
Subsystem | Report/Notes |
---|---|
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12942660/YARN-8842.004.patch |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22081/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 24s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. |
trunk Compile Tests | |||
+1 | mvninstall | 17m 27s | trunk passed |
+1 | compile | 0m 41s | trunk passed |
+1 | checkstyle | 0m 35s | trunk passed |
+1 | mvnsite | 0m 46s | trunk passed |
+1 | shadedclient | 11m 11s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 8s | trunk passed |
+1 | javadoc | 0m 26s | trunk passed |
Patch Compile Tests | |||
+1 | mvninstall | 0m 45s | the patch passed |
+1 | compile | 0m 38s | the patch passed |
+1 | javac | 0m 38s | the patch passed |
-0 | checkstyle | 0m 33s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 21 new + 142 unchanged - 2 fixed = 163 total (was 144) |
+1 | mvnsite | 0m 42s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 10m 59s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 1m 12s | the patch passed |
+1 | javadoc | 0m 23s | the patch passed |
Other Tests | |||
-1 | unit | 89m 9s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 25s | The patch does not generate ASF License warnings. |
137m 9s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12942681/YARN-8842.005.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 30b2dca12fea 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / cdf5d58 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22084/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22084/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22084/testReport/ |
Max. process+thread count | 886 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22084/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
According to our offline discussion with haibochen, here are the changes incorporated with the latest patch (patch006).
As I applied all the changes listed here as separate steps and ran the tests with every change, an unwanted code change is quite unlikely as the test were always passed.
1. Unified resource decrease methods into one single method, so that:
withContainersToDecrease, withVCoresToDecrease, withMemoryMBToDecrease, withCustomResToDecrease became withResourceToDecrease. This way, testcases are more concise and explicit.
2. Same as above, but for the declaration of resources and not decrease of resources.
2. Changed the QueueHierarchy class: instead of representing the hierarchy as a list of queue names, I created a Queue class that holds its name and a reference to its child queue. The validation code is also changed, an on-demand validation of queue names does happen when adding a child queue to the hierarchy.
3. The common parts of initializing QueueMetricsTestcase was extracted to a method.
4. Fixed test methods that only tested leaf queue metrics: now these methods get all the queue metrics and check the values on all queue levels. Example: testUpdatePreemptedSeconds previously only checked mqs.getLeafQueueSource, now it gets all the queue metrics sources and checks them.
5. Created separate classes for every type of custom resource: preempted seconds, allocated, available, pending, reserved.
Kept QueueMetricsForCustomResources class as a wrapper of all types, so that QueueMetrics should only refer to this class and it will delegate all alterations to the appropriate resource metric type objects.
Having an abstract class as a parent for the metric type classes named QueueMetricsCustomResourcesAbstract, I could keep the computation methods in one single place.
6. Added the performance optimization as wilfreds suggested: Custom resource metric update only happen if there is any custom resource defined.
Please note that:
1. The testcases in QueueMetricsForCustomResources would be more directly readable if the testXXX methods in QueueMetricsTestcase were moved to there. The reason why I don't do that is that QueueMetricsTestcase is intended to be a self-contained class that holds the data and the logic as well for all types of metrics testcases. Ideally, QueueMetricsTestcase could be used even from TestQueueMetrics, but I did not want to squeeze way too many changes into this patch.
2. I think it's not a right decision to move all metrics for cores / memory to QueueMetricsForCustomResources, as the former are annotated Metrics, the latter are not real metrics, but values stored in a map for every resource.
All in all, I like to idea to have all resource metrics in one class per type, but I would do that in a follow-up jira. One more thing to keep in mind is that AFAIK, the metric subsytem from Hadoop-common is not ready for using metrics from dynamic fields (e.g. Maps) so this will be quite tricky to implement.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 22s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 13s | Maven dependency ordering for branch |
+1 | mvninstall | 20m 22s | trunk passed |
+1 | compile | 8m 57s | trunk passed |
+1 | checkstyle | 1m 29s | trunk passed |
+1 | mvnsite | 1m 44s | trunk passed |
+1 | shadedclient | 15m 6s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 47s | trunk passed |
+1 | javadoc | 1m 36s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 25s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 32s | the patch passed |
+1 | compile | 12m 55s | the patch passed |
+1 | javac | 12m 55s | the patch passed |
-0 | checkstyle | 1m 41s | hadoop-yarn-project/hadoop-yarn: The patch generated 42 new + 185 unchanged - 2 fixed = 227 total (was 187) |
+1 | mvnsite | 2m 3s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 13m 57s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 35s | the patch passed |
+1 | javadoc | 1m 50s | the patch passed |
Other Tests | |||
+1 | unit | 3m 48s | hadoop-yarn-common in the patch passed. |
-1 | unit | 135m 55s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
231m 52s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceCapacity | |
hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower | |
hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect | |
hadoop.yarn.server.resourcemanager.TestResourceTrackerService | |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceFair | |
hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler | |
hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestApplicationMasterServiceWithFS | |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | |
hadoop.yarn.server.resourcemanager.TestRM | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId | |
hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate | |
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943135/YARN-8842.006.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 76d63121c2cd 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 6a39739 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22125/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22125/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22125/testReport/ |
Max. process+thread count | 970 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22125/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thank you snemeth. I am glad we gave this its own jira separate from the pre-emption changes. It has become quite large on its own.
The patch overall looks good. I have a couple of remarks:
- getMaxAllocationUtilization is new and not used outside of the patch, why would we add it if we're only going to leverage it in YARN-8059?
- With that simplification we also do not need the tests for getMaxAllocationUtilization in the TestQueueMetricsForCustomResources
- There was a possible NPE in getAllocatedResources until v7now fixed, combined with the fact that I do not understand the need for the isThereAnyAllocatedResource, the way the newInstance is created we can pass in an empty list or even a null. I would expect the method to look like this:
if (queueMetricsForCustomResources != null) { return Resource.newInstance(allocatedMB.value(), allocatedVCores.value(), queueMetricsForCustomResources.getAllocatedValues()); } return Resource.newInstance(allocatedMB.value(), allocatedVCores.value());
That does bring the QueueMetricsAllocatedCustomResources class back in line with all other custom resource metrics classes, just extend the abstract.
- Please look at the checkstyle issues mentioned.
A nit: on my system a number of indent inconsistencies showed up (mainly in TestQueueMetricsForCustomResources, QueueMetricsTestcase & TestQueueMetrics)
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 12s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 3m 22s | Maven dependency ordering for branch |
+1 | mvninstall | 22m 0s | trunk passed |
+1 | compile | 8m 34s | trunk passed |
+1 | checkstyle | 1m 29s | trunk passed |
+1 | mvnsite | 1m 46s | trunk passed |
+1 | shadedclient | 15m 10s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 1s | trunk passed |
+1 | javadoc | 1m 33s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 13s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 29s | the patch passed |
+1 | compile | 7m 24s | the patch passed |
+1 | javac | 7m 24s | the patch passed |
-0 | checkstyle | 1m 26s | hadoop-yarn-project/hadoop-yarn: The patch generated 37 new + 185 unchanged - 2 fixed = 222 total (was 187) |
+1 | mvnsite | 1m 48s | the patch passed |
+1 | whitespace | 0m 1s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 17s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 59s | the patch passed |
+1 | javadoc | 1m 25s | the patch passed |
Other Tests | |||
+1 | unit | 3m 18s | hadoop-yarn-common in the patch passed. |
-1 | unit | 122m 29s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. |
211m 35s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.server.resourcemanager.TestResourceTrackerService |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAppRunnability | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceCapacity | |
hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower | |
hadoop.yarn.server.resourcemanager.rmapp.TestApplicationLifetimeMonitor | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue | |
hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | |
hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceFair | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestSchedulingUpdate | |
hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens | |
hadoop.yarn.server.resourcemanager.resourcetracker.TestNMReconnect | |
hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem | |
hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter | |
hadoop.yarn.server.resourcemanager.TestRM | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppAttempt | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestApplicationMasterServiceWithFS | |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | |
hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerPreemption | |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerFairShare |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943154/YARN-8842.007.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux bbd710f84f59 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / edce866 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22127/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22127/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22127/testReport/ |
Max. process+thread count | 961 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22127/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 27s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 22s | Maven dependency ordering for branch |
+1 | mvninstall | 20m 16s | trunk passed |
+1 | compile | 9m 0s | trunk passed |
+1 | checkstyle | 1m 29s | trunk passed |
+1 | mvnsite | 1m 43s | trunk passed |
+1 | shadedclient | 14m 58s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 48s | trunk passed |
+1 | javadoc | 1m 26s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 13s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 24s | the patch passed |
+1 | compile | 8m 28s | the patch passed |
+1 | javac | 8m 28s | the patch passed |
-0 | checkstyle | 1m 25s | hadoop-yarn-project/hadoop-yarn: The patch generated 37 new + 185 unchanged - 2 fixed = 222 total (was 187) |
+1 | mvnsite | 1m 39s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 27s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 59s | the patch passed |
+1 | javadoc | 1m 23s | the patch passed |
Other Tests | |||
+1 | unit | 3m 16s | hadoop-yarn-common in the patch passed. |
-1 | unit | 94m 34s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 1m 4s | The patch does not generate ASF License warnings. |
181m 19s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943165/YARN-8842.008.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 36634744457f 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / edce866 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22129/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22129/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22129/testReport/ |
Max. process+thread count | 906 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22129/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
The test failure is not related to this change as I see it happening in different builds. Based on the comments on YARN-8468 Weiwei thinks it is related to that change.
Can you have a look at the comments I added, even after version 8 they are still relevant.
Hi wilfreds!
Thanks for coming back to this!
I think I addressed all your comments, with this answer: https://issues.apache.org/jira/browse/YARN-8842?focusedCommentId=16639171&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16639171
If you don't think so please list what is missing because I think all of them are addressed, except the getMaxAllocationUtilization method I haven't touched as I would prefer to do that with YARN-8059, which will be done immediately when this gets merged.
Thanks!
Based on our offline discussion with wilfreds, here are the changes included in patch009:
- Removed getMaxAllocationUtilization from QueueMetricsForCustomResources and its related testcases and moved the code to YARN-8059 patch002
- Removed the unnecessary check of queueMetricsForCustomResources.isThereAnyAllocatedResource in QueueMetrics#getAllocatedResources as Resource.newInstance already handles an empty-map.
- Fixed all the checkstyle issues, except "'xxx' hides a field. [HiddenField]" kind of issues. This could only disappear if the parameter has a different name than the fields, which does not make sense for me.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 7 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 25s | Maven dependency ordering for branch |
+1 | mvninstall | 20m 33s | trunk passed |
+1 | compile | 9m 16s | trunk passed |
+1 | checkstyle | 1m 30s | trunk passed |
+1 | mvnsite | 1m 44s | trunk passed |
+1 | shadedclient | 15m 3s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 53s | trunk passed |
+1 | javadoc | 1m 24s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 13s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 36s | the patch passed |
+1 | compile | 8m 17s | the patch passed |
+1 | javac | 8m 17s | the patch passed |
-0 | checkstyle | 1m 31s | hadoop-yarn-project/hadoop-yarn: The patch generated 15 new + 185 unchanged - 2 fixed = 200 total (was 187) |
+1 | mvnsite | 1m 50s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 27s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 8s | the patch passed |
+1 | javadoc | 1m 23s | the patch passed |
Other Tests | |||
+1 | unit | 3m 21s | hadoop-yarn-common in the patch passed. |
-1 | unit | 82m 49s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
170m 11s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943495/YARN-8842.009.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux afb0a586ff68 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2addebb |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22158/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22158/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22158/testReport/ |
Max. process+thread count | 866 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22158/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thank you for the updated patch.
I am OK with the patch as it is now v9, +1 non binding
There are two minor checkstyle issues which could be fixed on checkin:
- Class Queue should be declared as final. [FinalClass]
- Name 'conf' must match pattern '^[A-Z][A-Z0-9](_[A-Z0-9]+)$'. [ConstantName]
I cleaned up the patch a little bit.
Let's wait for jenkins with the results.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 29s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 12s | Maven dependency ordering for branch |
+1 | mvninstall | 19m 52s | trunk passed |
+1 | compile | 9m 7s | trunk passed |
+1 | checkstyle | 1m 30s | trunk passed |
+1 | mvnsite | 1m 48s | trunk passed |
+1 | shadedclient | 15m 32s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 55s | trunk passed |
+1 | javadoc | 1m 28s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 30s | the patch passed |
+1 | compile | 9m 4s | the patch passed |
+1 | javac | 9m 4s | the patch passed |
-0 | checkstyle | 1m 35s | hadoop-yarn-project/hadoop-yarn: The patch generated 27 new + 142 unchanged - 2 fixed = 169 total (was 144) |
+1 | mvnsite | 1m 41s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 12m 56s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 22s | the patch passed |
+1 | javadoc | 1m 28s | the patch passed |
Other Tests | |||
+1 | unit | 3m 32s | hadoop-yarn-common in the patch passed. |
-1 | unit | 87m 10s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 45s | The patch does not generate ASF License warnings. |
176m 4s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943734/YARN-8842.010.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux f5c545dfcab0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 28ca5c9 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22173/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22173/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22173/testReport/ |
Max. process+thread count | 871 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22173/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Hi haibochen!
Fixed some checkstyle issues with patch011.
I realized there are a bunch of issues related to fields are not private and they don't have accessor methods in test classes. I think for these classes it's perfectly fine to access them as package-private fields.
The other ones are for builder method parameters have the same name as the fields and they hide them: If I fixed this, it had no additional value and the param names could be too cryptic (abbrevations, for example).
What do you think, do we have to fix any of the outstanding checkstyle issues?
Thanks!
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 26s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 8s | Maven dependency ordering for branch |
+1 | mvninstall | 18m 11s | trunk passed |
+1 | compile | 7m 58s | trunk passed |
+1 | checkstyle | 1m 23s | trunk passed |
+1 | mvnsite | 1m 46s | trunk passed |
+1 | shadedclient | 14m 15s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 49s | trunk passed |
+1 | javadoc | 1m 29s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 13s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 26s | the patch passed |
+1 | compile | 7m 18s | the patch passed |
+1 | javac | 7m 18s | the patch passed |
-0 | checkstyle | 1m 19s | hadoop-yarn-project/hadoop-yarn: The patch generated 21 new + 142 unchanged - 2 fixed = 163 total (was 144) |
+1 | mvnsite | 1m 40s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 41s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 3s | the patch passed |
+1 | javadoc | 1m 25s | the patch passed |
Other Tests | |||
+1 | unit | 3m 37s | hadoop-yarn-common in the patch passed. |
-1 | unit | 89m 22s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
170m 21s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12943805/YARN-8842.011.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 53a570e520b0 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 5209c75 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22179/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22179/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22179/testReport/ |
Max. process+thread count | 937 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22179/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thanks snemeth for updating the patch! The patch looks much better to me. In addition to the checkstyle issues (I think we can ignore the ones like "HiddenField" and "must be private and have accessor methods"), I have one other minor comment: In QueueInfo, all fields can be marked as final I think. Let's also rename the methods following the pattern assert*() to check*(). +1 otherwise.
Hi haibochen!
I applied the changes you suggested, please check the latest patch!
Thanks!
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 20s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 1m 19s | Maven dependency ordering for branch |
+1 | mvninstall | 18m 2s | trunk passed |
+1 | compile | 8m 8s | trunk passed |
+1 | checkstyle | 1m 12s | trunk passed |
+1 | mvnsite | 1m 31s | trunk passed |
+1 | shadedclient | 13m 12s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 40s | trunk passed |
+1 | javadoc | 1m 14s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 14s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 23s | the patch passed |
+1 | compile | 7m 33s | the patch passed |
+1 | javac | 7m 33s | the patch passed |
-0 | checkstyle | 1m 13s | hadoop-yarn-project/hadoop-yarn: The patch generated 21 new + 142 unchanged - 2 fixed = 163 total (was 144) |
+1 | mvnsite | 1m 33s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 11m 37s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 1s | the patch passed |
+1 | javadoc | 1m 25s | the patch passed |
Other Tests | |||
+1 | unit | 3m 35s | hadoop-yarn-common in the patch passed. |
-1 | unit | 88m 26s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. |
167m 54s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944044/YARN-8842.012.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux c93c3e1523fa 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2614078 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22191/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22191/testReport/ |
Max. process+thread count | 935 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22191/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 22s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
trunk Compile Tests | |||
0 | mvndep | 3m 32s | Maven dependency ordering for branch |
+1 | mvninstall | 19m 45s | trunk passed |
+1 | compile | 9m 29s | trunk passed |
+1 | checkstyle | 1m 21s | trunk passed |
+1 | mvnsite | 1m 43s | trunk passed |
+1 | shadedclient | 14m 9s | branch has no errors when building and testing our client artifacts. |
+1 | findbugs | 2m 48s | trunk passed |
+1 | javadoc | 1m 22s | trunk passed |
Patch Compile Tests | |||
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 22s | the patch passed |
+1 | compile | 7m 26s | the patch passed |
+1 | javac | 7m 26s | the patch passed |
-0 | checkstyle | 1m 17s | hadoop-yarn-project/hadoop-yarn: The patch generated 21 new + 142 unchanged - 2 fixed = 163 total (was 144) |
+1 | mvnsite | 1m 34s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | shadedclient | 14m 46s | patch has no errors when building and testing our client artifacts. |
+1 | findbugs | 3m 1s | the patch passed |
+1 | javadoc | 1m 25s | the patch passed |
Other Tests | |||
+1 | unit | 3m 37s | hadoop-yarn-common in the patch passed. |
-1 | unit | 90m 13s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
179m 14s |
Subsystem | Report/Notes |
---|---|
Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12944044/YARN-8842.012.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 428b30eba13e 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 2614078 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_181 |
findbugs | v3.1.0-RC1 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/22193/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/22193/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/22193/testReport/ |
Max. process+thread count | 960 (vs. ulimit of 10000) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/22193/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
+1 on the latest patch. I'll fix the one minor indentation checkstyle issues at the time of my commit.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15232 (See https://builds.apache.org/job/Hadoop-trunk-Commit/15232/)
YARN-8842. Expose metrics for custom resource types in QueueMetrics. (haibochen: rev 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetricsForCustomResources.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueInfo.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceMetricsChecker.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/resourcetypes/ResourceTypesTestHelper.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java
- (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetrics.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetricsTestData.java
- (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestQueueMetricsForCustomResources.java
Thanks haibochen for the commit and for fixing the checkstyle indentation issue!
I backported to branch-3.2 and 3.1, but putting it into branch-2 seems to involve a tangled web of prerequisites.
Hi snemeth. I would like this functionality to be backported to branch-2. Lambda functions are used extensively in the trunk patch, which is a problem since we must still support Java 1.7 in branch-2.
Would you like to work on a branch-2 patch, or should I give it a try.
epayne regarding the tangled web of dependencies...I've ported YARN-7739, YARN-7541, YARN-8202 to branch-2. Now YARN-8842 seems to apply (mostly) cleanly.
The 1.7 support is not gonna be pretty though... There's a lot of 1.8 specific syntax in TestQueueMetricsForCustomResources and QueueInfo which passes methods into refactored functions. I can try to tackle this, let me know if you've already started work on this.
I've attached YARN-8842-branch-2.001.patch. Main differences are :
- Replacing expected resource maps with manually constructed ones, e.g: computeExpectedCustomResourceValues(testData.customResourceValues, (k, v) -> v * containers becomes
+ Map<String, Long> expected = new HashMap<>(); + for (Map.Entry<String, Long> entry : testData.customResourceValues.entrySet()) { + expected.put(entry.getKey(), entry.getValue() * containers); + }
- Duplicating all of the assertAllMetrics, assertQueueMetricsOnly, assertAllQueueMetrics, checkAllQueueMetrics four times, each one looks like assertAllXMetrics, assertQueueXMetricsOnly, assertAllXQueueMetrics, checkAllXQueueMetrics where X = {Pending, Allocated, Preempted, Reserved}
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 31m 16s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
branch-2 Compile Tests | |||
0 | mvndep | 0m 34s | Maven dependency ordering for branch |
+1 | mvninstall | 14m 50s | branch-2 passed |
+1 | compile | 9m 44s | branch-2 passed with JDK v1.7.0_95 |
+1 | compile | 8m 33s | branch-2 passed with JDK v1.8.0_222 |
+1 | checkstyle | 1m 8s | branch-2 passed |
+1 | mvnsite | 1m 58s | branch-2 passed |
+1 | findbugs | 3m 26s | branch-2 passed |
+1 | javadoc | 1m 59s | branch-2 passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 33s | branch-2 passed with JDK v1.8.0_222 |
Patch Compile Tests | |||
0 | mvndep | 0m 14s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 14s | the patch passed |
-1 | compile | 2m 16s | hadoop-yarn in the patch failed with JDK v1.7.0_95. |
-1 | javac | 2m 16s | hadoop-yarn in the patch failed with JDK v1.7.0_95. |
+1 | compile | 5m 43s | the patch passed with JDK v1.8.0_222 |
+1 | javac | 5m 43s | the patch passed |
-0 | checkstyle | 0m 52s | hadoop-yarn-project/hadoop-yarn: The patch generated 32 new + 139 unchanged - 2 fixed = 171 total (was 141) |
+1 | mvnsite | 1m 23s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 2m 39s | the patch passed |
-1 | javadoc | 0m 34s | hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) |
+1 | javadoc | 1m 5s | the patch passed with JDK v1.8.0_222 |
Other Tests | |||
-1 | unit | 3m 4s | hadoop-yarn-common in the patch failed. |
+1 | unit | 65m 3s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | asflicense | 0m 38s | The patch does not generate ASF License warnings. |
163m 53s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12985905/YARN-8842-branch-2.001.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 5e429cbae251 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | branch-2 / 235d6c7 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_222 |
Multi-JDK versions | /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_222 |
findbugs | v3.0.0 |
compile | https://builds.apache.org/job/PreCommit-YARN-Build/25174/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_95.txt |
javac | https://builds.apache.org/job/PreCommit-YARN-Build/25174/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_95.txt |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/25174/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/25174/artifact/out/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/25174/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25174/testReport/ |
Max. process+thread count | 838 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25174/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Unit test failure is related to YARN-7589, I'll port that after this Jira goes into branch-2.
YARN-8842-branch-2.002.patch fixes the javac/javadoc/compile error.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 42s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
branch-2 Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for branch |
+1 | mvninstall | 8m 40s | branch-2 passed |
+1 | compile | 6m 48s | branch-2 passed with JDK v1.7.0_95 |
+1 | compile | 5m 52s | branch-2 passed with JDK v1.8.0_222 |
+1 | checkstyle | 0m 52s | branch-2 passed |
+1 | mvnsite | 1m 27s | branch-2 passed |
+1 | findbugs | 2m 27s | branch-2 passed |
+1 | javadoc | 1m 20s | branch-2 passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 9s | branch-2 passed with JDK v1.8.0_222 |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 8s | the patch passed |
-1 | compile | 2m 18s | hadoop-yarn in the patch failed with JDK v1.7.0_95. |
-1 | javac | 2m 18s | hadoop-yarn in the patch failed with JDK v1.7.0_95. |
+1 | compile | 5m 29s | the patch passed with JDK v1.8.0_222 |
+1 | javac | 5m 29s | the patch passed |
-0 | checkstyle | 0m 51s | hadoop-yarn-project/hadoop-yarn: The patch generated 31 new + 139 unchanged - 2 fixed = 170 total (was 141) |
+1 | mvnsite | 1m 27s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 0s | the patch passed |
+1 | javadoc | 1m 32s | the patch passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 21s | the patch passed with JDK v1.8.0_222 |
Other Tests | |||
-1 | unit | 3m 17s | hadoop-yarn-common in the patch failed. |
+1 | unit | 64m 37s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
118m 44s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12985919/YARN-8842-branch-2.002.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e665b96675e9 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | branch-2 / 81060b3 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_222 |
Multi-JDK versions | /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_222 |
findbugs | v3.0.0 |
compile | https://builds.apache.org/job/PreCommit-YARN-Build/25176/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_95.txt |
javac | https://builds.apache.org/job/PreCommit-YARN-Build/25176/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_95.txt |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/25176/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/25176/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25176/testReport/ |
Max. process+thread count | 838 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25176/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Wow, thanks a lot, jhung!
I'll hopefully be able to get to the review early next week.
Thanks epayne! Uploading YARN-8842-branch-2.003.patch to fix another javac issue ...
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 56s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
branch-2 Compile Tests | |||
0 | mvndep | 0m 16s | Maven dependency ordering for branch |
+1 | mvninstall | 9m 39s | branch-2 passed |
+1 | compile | 6m 41s | branch-2 passed with JDK v1.7.0_95 |
+1 | compile | 5m 47s | branch-2 passed with JDK v1.8.0_222 |
+1 | checkstyle | 0m 52s | branch-2 passed |
+1 | mvnsite | 1m 33s | branch-2 passed |
+1 | findbugs | 2m 55s | branch-2 passed |
+1 | javadoc | 1m 27s | branch-2 passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 19s | branch-2 passed with JDK v1.8.0_222 |
Patch Compile Tests | |||
0 | mvndep | 0m 14s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 20s | the patch passed |
+1 | compile | 6m 43s | the patch passed with JDK v1.7.0_95 |
+1 | javac | 6m 43s | the patch passed |
+1 | compile | 5m 44s | the patch passed with JDK v1.8.0_222 |
+1 | javac | 5m 44s | the patch passed |
-0 | checkstyle | 0m 51s | hadoop-yarn-project/hadoop-yarn: The patch generated 30 new + 139 unchanged - 2 fixed = 169 total (was 141) |
+1 | mvnsite | 1m 25s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 3m 1s | the patch passed |
+1 | javadoc | 1m 17s | the patch passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 4s | the patch passed with JDK v1.8.0_222 |
Other Tests | |||
-1 | unit | 3m 2s | hadoop-yarn-common in the patch failed. |
-1 | unit | 64m 18s | hadoop-yarn-server-resourcemanager in the patch failed. |
+1 | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
124m 12s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
hadoop.yarn.server.resourcemanager.TestApplicationMasterService | |
hadoop.yarn.server.resourcemanager.scheduler.TestQueueMetricsForCustomResources | |
hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12985981/YARN-8842-branch-2.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux 5d382bd7580b 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | branch-2 / 6973f78 |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_222 |
Multi-JDK versions | /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_222 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/25182/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/25182/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/25182/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25182/testReport/ |
Max. process+thread count | 860 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25182/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
I think the additional hadoop-yarn-server-resourcemanager test failures were caused by YARN-7411 (which has since been reverted from branch-2: https://issues.apache.org/jira/browse/YARN-7411?focusedCommentId=16975442&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16975442)
Rerunning jenkins.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 44s | Docker mode activated. |
Prechecks | |||
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 6 new or modified test files. |
branch-2 Compile Tests | |||
0 | mvndep | 0m 24s | Maven dependency ordering for branch |
+1 | mvninstall | 12m 40s | branch-2 passed |
+1 | compile | 6m 50s | branch-2 passed with JDK v1.7.0_95 |
+1 | compile | 5m 47s | branch-2 passed with JDK v1.8.0_222 |
+1 | checkstyle | 0m 54s | branch-2 passed |
+1 | mvnsite | 1m 30s | branch-2 passed |
+1 | findbugs | 2m 29s | branch-2 passed |
+1 | javadoc | 1m 22s | branch-2 passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 8s | branch-2 passed with JDK v1.8.0_222 |
Patch Compile Tests | |||
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 6s | the patch passed |
+1 | compile | 6m 10s | the patch passed with JDK v1.7.0_95 |
+1 | javac | 6m 10s | the patch passed |
+1 | compile | 5m 41s | the patch passed with JDK v1.8.0_222 |
+1 | javac | 5m 41s | the patch passed |
-0 | checkstyle | 0m 51s | hadoop-yarn-project/hadoop-yarn: The patch generated 30 new + 139 unchanged - 2 fixed = 169 total (was 141) |
+1 | mvnsite | 1m 23s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | findbugs | 2m 37s | the patch passed |
+1 | javadoc | 1m 15s | the patch passed with JDK v1.7.0_95 |
+1 | javadoc | 1m 3s | the patch passed with JDK v1.8.0_222 |
Other Tests | |||
-1 | unit | 3m 2s | hadoop-yarn-common in the patch failed. |
+1 | unit | 63m 59s | hadoop-yarn-server-resourcemanager in the patch passed. |
+1 | asflicense | 0m 35s | The patch does not generate ASF License warnings. |
124m 49s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.yarn.api.TestPBImplRecords |
Subsystem | Report/Notes |
---|---|
Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:f555aa740b5 |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12985981/YARN-8842-branch-2.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e3e7e4a9a353 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/provided.sh |
git revision | branch-2 / a74411e |
maven | version: Apache Maven 3.3.9 |
Default Java | 1.8.0_222 |
Multi-JDK versions | /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 /usr/lib/jvm/java-8-openjdk-amd64:1.8.0_222 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/25184/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt |
unit | https://builds.apache.org/job/PreCommit-YARN-Build/25184/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/25184/testReport/ |
Max. process+thread count | 893 (vs. ulimit of 5500) |
modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/25184/console |
Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
Thanks for the patch, jhung.
+1
I will run some more tests and commit to branch-2 and branch-2.10 if no objections.
Thanks jhung and snemeth for all of your work on this JIRA. I committed YARN-8842-branch-2.003.patch to branch-2.
I would also like to port this back to branch-2.10. jhung, if you don't have any objections, I will cherry-pick YARN-7739, YARN-7541, YARN-8202, which are prerequi8sites.
I committed this and the aforementioned JIRAs to branch-2.10 as well.
Thanks for all your efforts, jhung
YARN-8842does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.YARN-8842This message was automatically generated.