Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Attachments
Attachments
- YARN-2619-1.patch
- 14 kB
- Wei Yan
- YARN-2619.002.patch
- 46 kB
- Varun Vasudev
- YARN-2619.003.patch
- 49 kB
- Varun Vasudev
- YARN-2619.004.patch
- 52 kB
- Varun Vasudev
Activity
YARN-3443 introduces a resource handler chain which cleans up a lot of the cgroup handling. I've modified the patch Wei uploaded to make it work with the new chain. This version of the patch just does a fair sharing of resources. Once we add support for scheduling, we can modify the weighting.
sidharta-s - can you please review?
Forgot to mention,
- The new patch refactors the tests added by Wei for the disk scheduling as well as the mtab parsing.
- Some of the modifications to CgroupsLCEResourceHandler are no longer required since
YARN-3443moves that work to a common class. - A new class has been added to handle the weight assigning for disk resources.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 32s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 4 new or modified test files. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 7m 32s | There were no new javac warning messages. |
+1 | javadoc | 9m 30s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 5m 31s | The applied patch generated 5 additional checkstyle issues. |
+1 | install | 1m 31s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 24s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 22s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 6m 6s | Tests passed in hadoop-yarn-server-nodemanager. |
48m 34s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12728463/YARN-2619.002.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 7f07c4d |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7512/artifact/patchprocess/checkstyle-result-diff.txt |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7512/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7512/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7512/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7512/console |
This message was automatically generated.
Thanks for the patch, vvasudev. Overall, the patch seems good - my only concern is about a dependency from TestCgroupsHandlerImpl to TestCgroupsLCEResourceHandler. More details inline:
YarnConfiguration.java
There seem to be a bunch of unrelated formatting changes in YarnConfiguration.java - consider removing these changes.
CGroupsHandler.java
public static final String CGROUP_FILE_TASKS = "tasks”; public static final String CGROUP_PARAM_CLASSID = "classid"; String CGROUP_PARAM_BLKIO = "weight”;
Make this string “public static final” to be consistent with the rest of the the strings here? It doesn’t seem like ‘weight’ is the only supported cgroup param for blkio. , consider re-naming accordingly ( e.g see classid above ). Alternatively, maybe rename both? CGROUP_PARAM_NET_CLS_CLASSID and CGROUP_PARAM_BLKIO_WEIGHT.
CGroupsHandlerImpl.java
private void init() throws ResourceHandlerException { initializeControllerPaths(); }
Extra empty line before this function.
@VisibleForTesting Map<CGroupController, String> getControllerPaths() { return controllerPaths; }
Private, modifiable state is being exposed here - use an unmodifiable map.
CGroupsLCEResourceHandler.java
@VisibleForTesting Map<CGroupController, String> getControllerPaths() { return controllerPaths; }
Same comment as above. Private, modifiable state is being exposed here - use an unmodifiable map.
Test CGroupsHandlerImpl.java
File parentDir = new File(tmpPath); // create mock cgroup File cpuCgroupMountDir = TestCgroupsLCEResourcesHandler.createMockCgroupMount(parentDir, "cpu", hierarchy); Assert.assertTrue(cpuCgroupMountDir.exists()); File blkioCgroupMountDir = TestCgroupsLCEResourcesHandler.createMockCgroupMount(parentDir, "blkio", hierarchy); Assert.assertTrue(blkioCgroupMountDir.exists()); File mockMtabFile = TestCgroupsLCEResourcesHandler.createMockMTab(parentDir); MyCgroupsHandler handler = new MyCgroupsHandler(conf, privilegedOperationExecutorMock);
IMO, it doesn’t make sense to have a dependency on TestCgroupsLCEResourceHandler here. Eventually, we want to get rid of CgroupsLCEResourceHandler (which is very CPU specific) and implement this using the new resource handler framework. CGroupsHandlerImpl is meant to be a replacement for (the cgroups functionality of) CgroupsLCEResourceHandler (See YARN-3542). Given that this is the case, we should consider implementing this functionality in TestCGroupsHandlerImpl.
This file also has some unrelated formatting changes.
@VisibleForTesting static void nullifyResourceHandlerChain() throws ResourceHandlerException {
Can this be made private instead of package-private? This shouldn’t be available for use except for testing.
CGroupsDiskResourceHandlerImpl.java
Class Name : Maybe we should use something less ‘generic’ - (with CFQ?) I expect that we’ll have more disk/cgroup resource handlers in place in the future - so a less generic name might be better here.
String[] lines = data.split("\n”);
Use %n or System.lineSeperator() ?
if (partition.startsWith("sd") || partition.startsWith("hd")) {
Is there a reason to only use these prefixes? I have seen vd* on some virtualized envs, for example.
PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP, "cgroups="
+ cGroupsHandler.getPathForCGroupTasks(
Use PrivilegedOperation.CGROUP_ARG_PREFIX instead ?
TestCGroupsDiskResourceHandlerImpl.java
Assert.assertEquals("cgroups=" + path, args.get(0));
Use PrivilegedOperation.CGROUP_ARG_PREFIX instead.
YarnConfiguration.java
There seem to be a bunch of unrelated formatting changes in YarnConfiguration.java - consider removing these changes.
CGroupsHandler.java
It fixes some checkstyle warnings about lines being more than 80 characters in length. Let me know if you feel strongly about it and I'll revert them.
public static final String CGROUP_FILE_TASKS = "tasks”;
public static final String CGROUP_PARAM_CLASSID = "classid";
String CGROUP_PARAM_BLKIO = "weight”;Make this string “public static final” to be consistent with the rest of the the strings here? It doesn’t seem like ‘weight’ is the only supported cgroup param for blkio. , consider re-naming accordingly ( e.g see classid above ). Alternatively, maybe rename both? CGROUP_PARAM_NET_CLS_CLASSID and CGROUP_PARAM_BLKIO_WEIGHT.
Changed to public static final and renamed to CGROUP_PARAM_BLKIO_WEIGHT.
CGroupsHandlerImpl.java
private void init() throws ResourceHandlerException {
initializeControllerPaths();
}Extra empty line before this function.
Fixed.
@VisibleForTesting
Map<CGroupController, String> getControllerPaths() {
return controllerPaths;
}Private, modifiable state is being exposed here - use an unmodifiable map.
CGroupsLCEResourceHandler.java@VisibleForTesting
Map<CGroupController, String> getControllerPaths() {
return controllerPaths;
}Same comment as above. Private, modifiable state is being exposed here - use an unmodifiable map.
Fixed both.
Test CGroupsHandlerImpl.java
File parentDir = new File(tmpPath);
// create mock cgroup
File cpuCgroupMountDir =
TestCgroupsLCEResourcesHandler.createMockCgroupMount(parentDir, "cpu",
hierarchy);
Assert.assertTrue(cpuCgroupMountDir.exists());
File blkioCgroupMountDir =
TestCgroupsLCEResourcesHandler.createMockCgroupMount(parentDir,
"blkio", hierarchy);
Assert.assertTrue(blkioCgroupMountDir.exists());
File mockMtabFile =
TestCgroupsLCEResourcesHandler.createMockMTab(parentDir);
MyCgroupsHandler handler =
new MyCgroupsHandler(conf, privilegedOperationExecutorMock);IMO, it doesn’t make sense to have a dependency on TestCgroupsLCEResourceHandler here. Eventually, we want to get rid of CgroupsLCEResourceHandler (which is very CPU specific) and implement this using the new resource handler framework. CGroupsHandlerImpl is meant to be a replacement for (the cgroups functionality of) CgroupsLCEResourceHandler (See
YARN-3542). Given that this is the case, we should consider implementing this functionality in TestCGroupsHandlerImpl.
This file also has some unrelated formatting changes.
Good point. Moved the functions to TestCGroupsHandlerImpl. The formatting changes are similar to the ones in YarnConfiguration - fixing some checkstyle warnings.
@VisibleForTesting
static void nullifyResourceHandlerChain() throws ResourceHandlerException {Can this be made private instead of package-private? This shouldn’t be available for use except for testing.
If it's made private, the tests can't reset the resource handler chain. The annotation isn't loaded into the JVM. If you have any other way to reset the chain, let me know.
CGroupsDiskResourceHandlerImpl.java
Class Name : Maybe we should use something less ‘generic’ - (with CFQ?) I expect that we’ll have more disk/cgroup resource handlers in place in the future - so a less generic name might be better here.
Changed it to CGroupsBlkioResourceHandler. Is that ok?
String[] lines = data.split("\n”);
Use %n or System.lineSeperator() ?
Fixed.
if (partition.startsWith("sd") || partition.startsWith("hd")) {
Is there a reason to only use these prefixes? I have seen vd* on some virtualized envs, for example.
Added vd and xvd prefix checks.
PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP, "cgroups="
+ cGroupsHandler.getPathForCGroupTasks(Use PrivilegedOperation.CGROUP_ARG_PREFIX instead ?
TestCGroupsDiskResourceHandlerImpl.javaAssert.assertEquals("cgroups=" + path, args.get(0));
Use PrivilegedOperation.CGROUP_ARG_PREFIX instead.
Fixed.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 56s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 4 new or modified test files. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 7m 43s | There were no new javac warning messages. |
+1 | javadoc | 9m 39s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 5m 27s | The applied patch generated 5 additional checkstyle issues. |
+1 | install | 1m 33s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 24s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 5m 50s | Tests passed in hadoop-yarn-server-nodemanager. |
49m 5s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12728769/YARN-2619.003.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 99fe03e |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7522/artifact/patchprocess/checkstyle-result-diff.txt |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7522/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7522/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7522/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7522/console |
This message was automatically generated.
vvasudev , the changes you made seem mine. There is one minor thing in the new patch - there is an unused import org.apache.hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler in TestCGroupsHandlerImpl . This can probably be removed at the time the commit goes in.
+1 to this patch.
vinodkv , can you please take a look at this patch?
A very clean patch!
This is essentially a rewrite. FYI ywskycn.
Seems like we are also not yet tying this for scheduling, that's my expectation too. But add a note on this JIRA as to how sharing happens? And the defaults (what does a weight of 500 mean?).
Should we deprecate LCEResourcesHandler hierarchy so that future work doesn't go there?
CGroupsDiskResourceHandlerImpl
- Add Override annotations for methods that are overridden so it is clear what behavior is dictated by the base interface
- Print a warning if PARTITIONS_FILE cannot be read?
The sharing is just an equal split of disk operations. The number 500 is just arbitrary - all that matters is that all containers get assigned the same weight which ensures they get an equal share of disk operations. Once we have scheduling support, the weight will be determined dynamically based on the allocated resources.
there is an unused import org.apache.hadoop.yarn.server.nodemanager.util.TestCgroupsLCEResourcesHandler in TestCGroupsHandlerImpl
Fixed.
And the defaults (what does a weight of 500 mean?)
Added a comment in the implementation that it's just arbitrary.
Should we deprecate LCEResourcesHandler hierarchy so that future work doesn't go there?
We should do it once YARN-3542 gets committed.
Add Override annotations for methods that are overridden so it is clear what behavior is dictated by the base interface
Fixed.
Print a warning if PARTITIONS_FILE cannot be read?
Fixed.
I've also re-factored part of CGroupsHandlerImpl to make testing cleaner(removing the need to read the controllerPaths map).
sidharta-s - can you please review to make sure the refactoring is ok?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 46s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 4 new or modified test files. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 8m 57s | There were no new javac warning messages. |
+1 | javadoc | 11m 4s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 4m 57s | The applied patch generated 5 additional checkstyle issues. |
+1 | install | 1m 43s | mvn install still works. |
+1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 23s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 22s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 6m 11s | Tests passed in hadoop-yarn-server-nodemanager. |
55m 21s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12729166/YARN-2619.004.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 8f82970 |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7541/artifact/patchprocess/checkstyle-result-diff.txt |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7541/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7541/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7541/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7541/console |
This message was automatically generated.
vvasudev , thanks for refactoring the test to be cleaner. The corresponding changes seem good to me.
Committed this to trunk and branch-2. Thanks Varun!
Tx to Wei for the initial patch and the test-cases. And Sid for the reviews!
SUCCESS: Integrated in Hadoop-trunk-Commit #7711 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7711/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #180 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/180/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
FAILURE: Integrated in Hadoop-Yarn-trunk #914 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/914/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
FAILURE: Integrated in Hadoop-Hdfs-trunk #2112 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2112/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #171 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/171/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #181 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/181/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2130 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2130/)
YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. (vinodkv: rev 1b3b9e5c31c38388c1ce4208c65e8dd5f956da82)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/DiskResourceHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsBlkioResourceHandlerImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/CHANGES.txt
I really would have liked to review or at least skim through the patch before it went in. I should have been watching this issue, but would have appreciated a ping especially given the patrons were aware of my involvement on YARN-2139 and YARN-2168.
Also, given this is a sub-task of YARN-2139 and we have decided on doing it on a branch, I am surprised we committed this to trunk and branch-2. Is it too late to move it to the branch so we can look at all disk IO together?
I pinged Wei above and waited for a few days. I should have pinged you but missed, my bad.
It wasn't originally clear to me that this was going to a branch. Hence the confusion I expressed over at YARN-2618.
What's done is done. Clearly the patch here made sure everything is private, so I wouldn't be worried about removing it from trunk/branch-2. So I propose we create a new branch for YARN-2139 and proceed work there from this point of time.
Prelim patch for reviewing. The patch is based on YARN-2618.