Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 2.7.1
-
Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
-
Component/s: capacityscheduler
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
CapacityScheduler's LeafQueue has "reservations continue looking" logic that allows an application to unreserve elsewhere to fulfil a container request on a node that has available space. However in 2.7 that logic seems to break allocations for subsequent apps in the queue. Once a user hits its user limit, subsequent apps in the queue for other users receive containers at a significantly reduced rate.
Issue Links
- is broken by
-
YARN-3434
Interaction between reservations and userlimit can result in significant ULF violation
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Patch that resets the amount needed to unreserve at the beginning of canAssignToUser. That way subsequent users in the loop will not accidentally inherit a previous user's amount.
A potential workaround until this appears in a release is to set yarn.scheduler.capacity.reservations-continue-look-all-nodes to false in capacity-scheduler.xml. Note that this property is refreshable via yarn rmadmin -refreshQueues, so changing it does not require a restart. After this fix the property should be restored to true to avoid the original issue fixed in YARN-3434.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 11m 53s | trunk passed |
| +1 | compile | 0m 57s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 0m 48s | trunk passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 23s | trunk passed |
| +1 | mvnsite | 0m 58s | trunk passed |
| +1 | mvneclipse | 0m 24s | trunk passed |
| +1 | findbugs | 1m 50s | trunk passed |
| +1 | javadoc | 0m 43s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 0m 43s | trunk passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 48s | the patch passed |
| +1 | compile | 0m 49s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 0m 49s | the patch passed |
| +1 | compile | 0m 45s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 0m 45s | the patch passed |
| -1 | checkstyle | 0m 20s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 55 unchanged - 1 fixed = 56 total (was 56) |
| +1 | mvnsite | 0m 49s | the patch passed |
| +1 | mvneclipse | 0m 20s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 6s | the patch passed |
| +1 | javadoc | 0m 37s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 0m 40s | the patch passed with JDK v1.7.0_91 |
| -1 | unit | 73m 39s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 68m 17s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. |
| +1 | asflicense | 0m 23s | Patch does not generate ASF License warnings. |
| 188m 9s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationPriority |
| hadoop.yarn.server.resourcemanager.TestClientRMTokens | |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| hadoop.yarn.server.resourcemanager.TestClientRMService | |
| hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler | |
| JDK v1.7.0_91 Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization |
This message was automatically generated.
Thanks Jason Lowe, it's a nice finding!
+1 to the fix, but could you take a look at failed tests? Not sure if they're related to this fix.
+1. Thanks for fixing this.
could you take a look at failed tests? Not sure if they're related to this fix.
The TestApplicationPriority failure appears to be unrelated. I tried multiple times to reproduce it with the patch applied, and it always passes.
The TestClientRMTokens failure is unrelated. It's failing the same way for other precommit builds, and is being tracked by YARN-4306 / HADOOP-12687.
The TestAMAuthorization failure is unrelated. It's failing the same way for other precommit builds, and is being tracked by YARN-4318 / HADOOP-12687.
The TestClientRMService failure is unrelated. It's failing the same way in some other precommit builds (e.g.: see https://builds.apache.org/job/PreCommit-YARN-Build/10285/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt). I tried running it multiple times with the patch applied, and it always passed. I couldn't find a tracking ticket for this one, so I filed YARN-4613.
I'm guessing the TestAbstractYarnScheduler is unrelated. Note that it didn't fail with JDK7 but does with JDK8. I tried isolating the test with JDK8, and it doesn't fail, yet it fails when run as part of the suite. I'll investigate it further to see if I can narrow down why it's failing in JDK8, but the fact that it's passing on another JDK and also passes when run in isolation indicates it's likely not a problem with the patch.
Forgot to mention that TestApplicationPriority is failing in the same way for other precommit builds (e.g.: https://builds.apache.org/job/PreCommit-YARN-Build/10177/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt), and only failed on one JDK version vs. another. Couldn't find a tracking ticket for that one, so I filed YARN-4614.
TestAbstractYarnScheduler is failing the same way occasionally on trunk without the patch applied. Filed YARN-4615.
Ok thanks for investigating. +1 from me feel free to commit.
Thanks for the reviews! Committing this.
Actually it doesn't apply to branch-2.7, so working on a patch for that as well.
Patch for branch-2.7. Clearing of the amount needed to unreserve had to be in a different place for 2.7 since there were multiple places that could set that value (i.e.: canAssignToThisQueue also sets it).
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | docker | 0m 7s | Dockerfile '/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/dev-support/docker/Dockerfile' not found, falling back to built-in. |
| -1 | docker | 13m 55s | Docker failed to build yetus/hadoop:date2016-01-20. |
| Subsystem | Report/Notes |
|---|---|
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12783421/YARN-4610.branch-2.7.001.patch |
| JIRA Issue | |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/10345/console |
This message was automatically generated.
Sorry after looking some more I think there might be an issue with this for parent queue max capacities, looking some more.
Nice catch, Tom! I think we need to preserve the original amount needed to unreserve and reset back to that before evaluating each application. That way we'll honor the amount needed that can be passed in from the parent. Updated the branch-2.7 patch accordingly.
It's always zero from the parent in trunk so no need to preserve it, but I can update the trunk patch to match if desired.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 8m 17s | branch-2.7 passed |
| +1 | compile | 0m 31s | branch-2.7 passed with JDK v1.8.0_66 |
| +1 | compile | 0m 28s | branch-2.7 passed with JDK v1.7.0_91 |
| +1 | checkstyle | 0m 28s | branch-2.7 passed |
| +1 | mvnsite | 0m 36s | branch-2.7 passed |
| +1 | mvneclipse | 0m 17s | branch-2.7 passed |
| -1 | findbugs | 1m 15s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in branch-2.7 has 1 extant Findbugs warnings. |
| +1 | javadoc | 0m 24s | branch-2.7 passed with JDK v1.8.0_66 |
| +1 | javadoc | 0m 25s | branch-2.7 passed with JDK v1.7.0_91 |
| +1 | mvninstall | 0m 29s | the patch passed |
| +1 | compile | 0m 24s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 0m 24s | the patch passed |
| +1 | compile | 0m 27s | the patch passed with JDK v1.7.0_91 |
| +1 | javac | 0m 27s | the patch passed |
| -1 | checkstyle | 0m 27s | hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 1 new + 824 unchanged - 0 fixed = 825 total (was 824) |
| +1 | mvnsite | 0m 32s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| -1 | whitespace | 0m 3s | The patch has 1617 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| -1 | whitespace | 0m 47s | The patch has 98 line(s) with tabs. |
| +1 | findbugs | 1m 19s | the patch passed |
| +1 | javadoc | 0m 17s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 0m 23s | the patch passed with JDK v1.7.0_91 |
| -1 | unit | 53m 11s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 53m 55s | hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_91. |
| -1 | asflicense | 45m 5s | Patch generated 77 ASF License warnings. |
| 172m 0s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| hadoop.yarn.server.resourcemanager.TestClientRMTokens | |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization | |
| JDK v1.7.0_91 Failed junit tests | hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
| hadoop.yarn.server.resourcemanager.TestClientRMTokens | |
| hadoop.yarn.server.resourcemanager.TestAMAuthorization |
This message was automatically generated.
Whitespace complaints are bogus, and test failures are unrelated. The TestClientRMTokens and TestAMAuthorization failures are the same as in trunk. TestSystemMetricsPublisher is failing on branch-2.7 without the patch, filed YARN-4623.
+1 for branch 2.7. After investigating this some more the original patch of setting it to none() works. The reason is that the parents limit is passed and it would be taken into account int he leaf calculation. I think the latter patch is safer but either is fine with me.
The master patch I'm not sure about how its taking the max capacity into account so I'll have to look at that more, but the unit tests are passing and that would be a separate issue from this fix. +1 on that patch as well.
Thanks for the reviews! Committing this.
Thanks to Wangda and Tom for reviews! I committed this to trunk, branch-2, branch-2.8, and branch-2.7.
FAILURE: Integrated in Hadoop-trunk-Commit #9153 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9153/)
YARN-4610. Reservations continue looking for one app causes other apps (jlowe: rev 468a53b22f4ac5bb079dff986ba849a687d709fe)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
Closing the JIRA as part of 2.7.3 release.
I believe the issue is in LeafQueue#assignToUser. That method will modify the amount needed to unreserve for a particular user when they hit the resource limit. However the amount needed to unreserve never gets reset to zero for the next iteration of the loop, so subsequent apps for different users can end up not receiving containers because it accidentally thinks it needs to unreserve based on that stale value.