Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.7.0
-
None
-
Reviewed
Description
YARN should be aware of the resource utilization of the nodes when scheduling containers. For this, this task will implement the collection of memory/cpu usage on the node.
Attachments
Attachments
- YARN-3534-1.patch
- 21 kB
- Íñigo Goiri
- YARN-3534-2.patch
- 39 kB
- Íñigo Goiri
- YARN-3534-3.patch
- 39 kB
- Íñigo Goiri
- YARN-3534-3.patch
- 39 kB
- Íñigo Goiri
- YARN-3534-4.patch
- 9 kB
- Íñigo Goiri
- YARN-3534-5.patch
- 29 kB
- Íñigo Goiri
- YARN-3534-6.patch
- 9 kB
- Íñigo Goiri
- YARN-3534-7.patch
- 9 kB
- Íñigo Goiri
- YARN-3534-8.patch
- 9 kB
- Íñigo Goiri
- YARN-3534-9.patch
- 8 kB
- Íñigo Goiri
- YARN-3534-10.patch
- 15 kB
- Íñigo Goiri
- YARN-3534-11.patch
- 17 kB
- Íñigo Goiri
- YARN-3534-12.patch
- 18 kB
- Íñigo Goiri
- YARN-3534-14.patch
- 11 kB
- Íñigo Goiri
- YARN-3534-15.patch
- 12 kB
- Íñigo Goiri
- YARN-3534-16.patch
- 15 kB
- Íñigo Goiri
- YARN-3534-16.patch
- 15 kB
- Íñigo Goiri
- YARN-3534-17.patch
- 15 kB
- Íñigo Goiri
- YARN-3534-17.patch
- 15 kB
- Íñigo Goiri
- YARN-3534-18.patch
- 16 kB
- Íñigo Goiri
Issue Links
- breaks
-
YARN-4330 MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.
- Resolved
- is depended upon by
-
YARN-3980 Plumb resource-utilization info in node heartbeat through to the scheduler
- Resolved
-
YARN-4055 Report node resource utilization in heartbeat
- Resolved
- is related to
-
HADOOP-12210 Collect network usage on the node
- Resolved
- is required by
-
YARN-3481 Report NM aggregated container resource utilization in heartbeat
- Resolved
- relates to
-
YARN-545 NodeResourceMonitor and its Impl are emty and may be removed
- Resolved
Activity
Sunil G, my plan is to send the same entity as in YARN-3482: ResourceUtilization. For now, it will have memory in MB and Virtual Cores as a float. We can extend this for other resources.
In terms of the collection of the utilization, this will be monitored by the NodeResourceMonitor (currently empty) which will mimic the behavior of the ContainerMonitor which uses two configuration parameters: (1) if we monitor the resources of the node and (2) how often the resources are monitored.
I'm in the last stages of testing it and tomorrow I'll post a patch with this change. The structure was already there so it's a matter of adding the communication part..
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
-1 | patch | 0m 0s | The patch command could not apply the patch during dryrun. |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12727762/YARN-3534-1.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 0b3f895 |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7484/console |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 35s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
-1 | whitespace | 0m 0s | The patch has 78 line(s) that end in whitespace. |
+1 | javac | 7m 29s | There were no new javac warning messages. |
+1 | javadoc | 9m 34s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 5m 32s | The applied patch generated 11 additional checkstyle issues. |
+1 | install | 1m 33s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 40s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 21s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 1s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-server-common. |
+1 | yarn tests | 5m 55s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 2s |
This message was automatically generated.
Removing spaces at the end of the line; Jenkins doesn't seem to like them anymore.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 38s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 7m 29s | There were no new javac warning messages. |
+1 | javadoc | 9m 34s | There were no new javadoc warning messages. |
+1 | release audit | 0m 21s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 7m 47s | The applied patch generated 11 additional checkstyle issues. |
+1 | install | 1m 32s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 37s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 29s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 2m 1s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-server-common. |
+1 | yarn tests | 6m 5s | Tests passed in hadoop-yarn-server-nodemanager. |
55m 33s |
This message was automatically generated.
Skimmed through the latest patch. High-level comments/questions:
- Do we need a separate class/proto for ResourceUtilization? Could we just reuse Resource? That should make the patch significantly small.
- Would be nice to have NodeResourceMonitor emit metrics for usage. We could do this on a follow-up JIRA.
I notice the patch tries to provide utilization information in bytes for memory and float for CPU. Since the RM schedules in MB and vcores, seeing the utilization as rounded-up values in a Resource object is probably enough.
The original reason for having ResourceUtilization was getting better granularity for the CPUs. We had some discussion about it in YARN-3481; take a look there and chim in.
My original implementation had Resource but when trying to do scheduling of containers based on this, there were a lot of holes in the scheduling. Given this, I thought this patch was a good place to create this new utilization entity with CPU as a float.
Regarding the metrics in the NodeResourceMonitor, I completely agree. I thought about doing it right away but as you mentioned, it seemed a better idea to save it for another JIRA. Let's do that.
Thanks for pointing me to YARN-3481, Inigo. Didn't see the other JIRA.
Reading through those comments, I agree it might be better to add a separate class/proto for ResourceUtilization. Efforts like Tetris could add disk/network utilization even if the API itself doesn't support them explicitly.
Please see my comment on YARN-3481.
If nothing, we should separate the reporting bits from the measurement bit?
Also, we don't need the ResourceUtilizationCalculator layers? Once the information reaches the ResourceManager, RM can use the regular Resources utils.
I could first do the NodeResourceMonitorImpl and then add that info it to the heartbeat in a second JIRA. However, I thought the addition to the heartbeat was simple enough. Anyways, up to you; I can split it in two.
Regarding the extra ResourceUtilizations layers, I could get rid of them for now as it's not currently needed. However, in parallel, I'm privately working on a scheduler which is aware of utilization and it's much better to use those layers as I need the floats for the CPUs instead of plain int. I thought about making ResourceUtilizaiton an extension of Resource but it does not work...
Anyway, I'll send a patch without the extra ResourceUtilizations layer as it's not really required for this patch. Does that sound reasonable?
Thanks for the path Íñigo Goiri! Is it possible for you to split up the patch into two - one to record the memory and cpu utilization in NodeResourceMonitorImpl and one for the RPC changes. If you could file a separate JIRA for the NodeResourceMonitorImpl changes that you have made(they are definitely required and should go in), that would be ideal.
I'm not sure if we should use RPC to expose the stats. Using RPC restricts the stats to a limited number of services whereas using REST lets a larger number of services access the stats - which is why YARN-3332 wants to use REST to expose the NM stats. Using REST should also make it easier to add more stats with respect to utilization.
I'm not sure if we should use RPC to expose the stats.
As YARN-3332 suggests, we should expose these stats through REST. However, for the RM (scheduler) to get this information, I think RPC (through node heartbeat) is better. That way, the RM doesn't have to poll the REST API for every node there is an NM heartbeat for.
If the suggestion is for the NM to get this information from the REST API, I think that is fine. However, since this communication is private to NM, we can always change this later once YARN-3332 is ready.
It looks like there's a consensus on splitting on monitoring and reporting. If everybody is cool with it, I will rename this JIRA to "Collect node resource utilization" and just do the implementaiton of NodeResourceMonitor.
The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?
Isolating the changes for the NodeResourceMonitorImpl. The aggregationg of data and sending to the RM will be done in another JIRA.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 33s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
-1 | javac | 1m 48s | The patch appears to cause the build to fail. |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12729617/YARN-3534-4.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / f0db797 |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7560/console |
This message was automatically generated.
A patch to test-patch or smart-apply-patch has been detected.
Re-executing against the patched versions to perform further tests.
The console is at https://builds.apache.org/job/PreCommit-YARN-Build/7564/console in case of problems.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | dev-support patch detected. |
0 | pre-patch | 14m 45s | Pre-patch trunk compilation is healthy. |
0 | @author | 0m 0s | Skipping @author checks as test-patch has been patched. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | javac | 6m 29s | There were no new javac warning messages. |
+1 | javadoc | 7m 25s | There were no new javadoc warning messages. |
-1 | release audit | 22m 41s | The applied patch generated 364 release audit warnings. |
-1 | checkstyle | 4m 26s | The applied patch generated 2 additional checkstyle issues. |
0 | shellcheck | 4m 26s | Shellcheck was not available. |
+1 | install | 1m 12s | mvn install still works. |
+1 | eclipse:eclipse | 0m 30s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 4s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | common tests | 23m 7s | Tests passed in hadoop-common. |
+1 | yarn tests | 0m 29s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 5m 47s | Tests failed in hadoop-yarn-server-nodemanager. |
90m 58s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService |
This message was automatically generated.
Íñigo Goiri - the latest patch(YARN-3534-5.patch) has modifications to the checkstyle script as well as OsSecureRandom.java. I assume those changes are not meant to be part of your patch. Can you please upload a patch with just your changes? Thanks!
Sorry about that, I did the diff and then I did the patch 10 minutes after... meanwhile there were change sin trunk and it got all messed up. I think this one should be OK. No unit tests though.
The patch artifact directory has been removed!
This is a fatal error for test-patch.sh. Aborting.
Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7569/ may provide some hints.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 42s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | javac | 7m 33s | There were no new javac warning messages. |
+1 | javadoc | 9m 36s | There were no new javadoc warning messages. |
+1 | release audit | 0m 21s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 22s | The applied patch generated 2 new checkstyle issues (total was 211, now 213). |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 35s | mvn install still works. |
+1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 24s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 5m 48s | Tests failed in hadoop-yarn-server-nodemanager. |
44m 33s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12729860/YARN-3534-7.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / 6f541ed |
checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7589/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7589/console |
This message was automatically generated.
Fixing code style issues (nwo the output of the checker is meaningful so life is much easier).
The broken tests weren't related to my changes.
Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 14m 39s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
-1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
+1 | javac | 7m 31s | There were no new javac warning messages. |
+1 | javadoc | 9m 30s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
+1 | checkstyle | 1m 28s | There were no new checkstyle issues. |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 33s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 23s | The patch does not introduce any new Findbugs (version 2.0.3) warnings. |
+1 | yarn tests | 0m 30s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 5m 51s | Tests passed in hadoop-yarn-server-nodemanager. |
44m 26s |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12729911/YARN-3534-8.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / f1a152c |
hadoop-yarn-api test log | https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-api.txt |
hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/7594/testReport/ |
Java | 1.7.0_55 |
uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/7594/console |
This message was automatically generated.
The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?
I believe there are existing tickets for this. I started consolidating all the related efforts, let me do that so that you can start working on this once I identify the right JIRA.
Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.
You can reuse some of the code at TestLinuxResourceCalculatorPlugin and write a new TestNodeResourceMonitor that measures utilization against mock information.
Couple other comments:
- context variable is unused
- Can we also capture vmem usage as well as cpu-wall time?
- As per my proposal at YARN-3332, I'd like to refactor this later to be the statistics-agent. So, for now, can you just move the configuration-properties from YarnConfiguration into NodeResourceMonitorImpl?
Removed nodemanager context.
Added vmem (I cannot put it inside the Resource though).
Where do you want to put the vmem and where to get the cpu-wall time?
Regarding YarnConfiguration, I didn't understand what you wanted to do there. Do you want to put the constants in the class itself?
Thanks for working on this, Inigo.
Few comments:
- Given these stats are to be sent to the RM in heartbeat, should we capture the aggregate node resource usage in ResourceUtilization?
- Instead of adding a separate DEFAULT_NM_NODE_MON_INTERVAL_MS, we should probably just re-use the default for container-monitor?
- Also, should we add another config yarn.nodemanager.usage-monitor.interval-ms that both container-monitor and node-monitor inherit unless specified otherwise? If that seems reasonable, we should deprecate the default value for container-monitor-interval.
- For the monitoring thread,
- set the thread name?
- make it a daemon thread?
- on monitoringThread.join(), specify a timeout as well.
- in the corresponding catch-block, at least log that we couldn't wait until the monitoring-thread is interrupted.
I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
-1 | pre-patch | 16m 42s | Findbugs (version ) appears to be broken on trunk. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
-1 | javac | 7m 34s | The applied patch generated 1 additional warning messages. |
-1 | javadoc | 9m 39s | The applied patch generated 3 additional warning messages. |
-1 | release audit | 0m 18s | The applied patch generated 1 release audit warnings. |
-1 | checkstyle | 1m 24s | The applied patch generated 9 new checkstyle issues (total was 212, now 220). |
-1 | whitespace | 0m 1s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
+1 | install | 1m 36s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 18s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 21s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 4s | Tests passed in hadoop-yarn-server-nodemanager. |
50m 53s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 18s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | javac | 7m 32s | There were no new javac warning messages. |
+1 | javadoc | 9m 39s | There were no new javadoc warning messages. |
-1 | release audit | 0m 18s | The applied patch generated 1 release audit warnings. |
-1 | checkstyle | 1m 44s | The applied patch generated 1 new checkstyle issues (total was 212, now 211). |
+1 | whitespace | 0m 1s | The patch has no lines that end in whitespace. |
+1 | install | 1m 34s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 15s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 21s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 13s | Tests passed in hadoop-yarn-server-nodemanager. |
52m 52s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 30s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
+1 | javac | 7m 34s | There were no new javac warning messages. |
+1 | javadoc | 9m 39s | There were no new javadoc warning messages. |
-1 | release audit | 0m 19s | The applied patch generated 1 release audit warnings. |
-1 | checkstyle | 1m 41s | The applied patch generated 1 new checkstyle issues (total was 213, now 212). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 34s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 20s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 3s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 3s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 17m 13s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
-1 | javac | 2m 59s | The patch appears to cause the build to fail. |
Subsystem | Report/Notes |
---|---|
Patch URL | http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch |
Optional Tests | javadoc javac unit findbugs checkstyle |
git revision | trunk / ac60483 |
Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8483/console |
This message was automatically generated.
Fixed issue with constants.
We need to agree that these names are OK and that we are fine with deprecating the old ones.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 17m 26s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 1 new or modified test files. |
-1 | javac | 7m 46s | The applied patch generated 2 additional warning messages. |
+1 | javadoc | 9m 37s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 19s | The applied patch generated 5 new checkstyle issues (total was 211, now 215). |
+1 | whitespace | 0m 0s | The patch has no lines that end in whitespace. |
+1 | install | 1m 24s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 2m 44s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 25s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 6m 6s | Tests failed in hadoop-yarn-server-nodemanager. |
48m 0s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService |
hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService | |
hadoop.yarn.server.nodemanager.containermanager.container.TestContainer | |
hadoop.yarn.server.nodemanager.TestDeletionService |
This message was automatically generated.
Thanks for updating the patch, Íñigo Goiri. The latest patch looks pretty good. Few minor comments:
- Let us not deprecate NM_CONTAINER_MON_INTERVAL_MS. Deprecating the default value seems reasonable. One might want to specify a different interval for node monitoring and container monitoring.
- Make corresponding changes to yarn-site.xml
- NodeResourceMonitorImpl#serviceStop: the log message should say couldn't wait for the thread to join, and may be we should log it as WARN instead of INFO
- ContainersMonitorImpl: Users could continue to use the deprecated configs. If those are specified and the new ones are not, we should fallback to the deprecated ones. If this becomes too complicated, I am comfortable with leaving the configs for container-monitoring as is. We could file a follow-up JIRA to clean these configs for Hadoop-3.x.
I guess in #2 you meant yarn-default.xml, right?
I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 46s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 3 new or modified test files. |
+1 | javac | 7m 40s | There were no new javac warning messages. |
+1 | javadoc | 9m 40s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 46s | The applied patch generated 4 new checkstyle issues (total was 211, now 214). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 23s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 20s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 9s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 43s |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 36s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 3 new or modified test files. |
+1 | javac | 7m 37s | There were no new javac warning messages. |
+1 | javadoc | 9m 38s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 48s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
-1 | checkstyle | 2m 33s | The applied patch generated 2 new checkstyle issues (total was 28, now 27). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 20s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 22s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
+1 | yarn tests | 1m 56s | Tests passed in hadoop-yarn-common. |
+1 | yarn tests | 6m 10s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 36s |
This message was automatically generated.
Hopefully, last set of comments:
- We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should have been more explicit in my previous comment.
- ContainersMonitorImpl should prioritize container-specific configs and fallback if they are not present. Something along these lines:
this.monitoringInterval = conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS, conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS, YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS)); Class<? extends ResourceCalculatorPlugin> clazz = conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR, conf.getClass( YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null, ResourceCalculatorPlugin.class), ResourceCalculatorPlugin.class);
- yarn-default.xml needs to be updated for the CONTAINER_MON configs we are changing the behavior of, and any default values there should be removed.
- I missed the Deprecated for the calculator... my bad. Fixed.
- That's good; fixed.
- The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though.
Incorporated Karthik Kambatla improvements but still not clear on the yarn-default.xml.
In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used."
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 45s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 3 new or modified test files. |
+1 | javac | 7m 41s | There were no new javac warning messages. |
+1 | javadoc | 9m 39s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 45s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
+1 | whitespace | 0m 3s | The patch has no lines that end in whitespace. |
+1 | install | 1m 22s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 22s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 1m 54s | Tests failed in hadoop-yarn-common. |
-1 | yarn tests | 6m 7s | Tests failed in hadoop-yarn-server-nodemanager. |
53m 35s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.util.TestRackResolver |
hadoop.yarn.server.nodemanager.TestDeletionService |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 52s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 3 new or modified test files. |
+1 | javac | 7m 45s | There were no new javac warning messages. |
+1 | javadoc | 9m 40s | There were no new javadoc warning messages. |
+1 | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 51s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 22s | mvn install still works. |
+1 | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 20s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 23s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 1m 55s | Tests failed in hadoop-yarn-common. |
+1 | yarn tests | 6m 8s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 54s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.util.TestRackResolver |
This message was automatically generated.
The latest patches (v17) don't seem to have the updates. Íñigo Goiri - could you please verify?
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | pre-patch | 18m 35s | Pre-patch trunk compilation is healthy. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | tests included | 0m 0s | The patch appears to include 3 new or modified test files. |
+1 | javac | 7m 39s | There were no new javac warning messages. |
+1 | javadoc | 9m 40s | There were no new javadoc warning messages. |
+1 | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. |
-1 | checkstyle | 1m 46s | The applied patch generated 1 new checkstyle issues (total was 211, now 211). |
+1 | whitespace | 0m 2s | The patch has no lines that end in whitespace. |
+1 | install | 1m 23s | mvn install still works. |
+1 | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. |
+1 | findbugs | 4m 21s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. |
+1 | yarn tests | 0m 24s | Tests passed in hadoop-yarn-api. |
-1 | yarn tests | 1m 55s | Tests failed in hadoop-yarn-common. |
+1 | yarn tests | 6m 10s | Tests passed in hadoop-yarn-server-nodemanager. |
53m 33s |
Reason | Tests |
---|---|
Failed unit tests | hadoop.yarn.util.TestRackResolver |
This message was automatically generated.
Just committed to trunk and branch-2.
Thanks Íñigo Goiri for the contribution and patience through the reviews.
FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8311/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
Thank you Karthik Kambatla for taking care of the review and the commits.
I'll be moving to propagating this info to the scheduler.
FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/)
YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
Íñigo Goiri, Karthik Kambatla, could you comment on extending this to also take in network and disk usage information?
Is this going to be configurable item to look for, or all resources will be monitored?