Details

    • Reviewed

    Description

      YARN should be aware of the resource utilization of the nodes when scheduling containers. For this, this task will implement the collection of memory/cpu usage on the node.

      Attachments

        1. YARN-3534-1.patch
          21 kB
          Íñigo Goiri
        2. YARN-3534-2.patch
          39 kB
          Íñigo Goiri
        3. YARN-3534-3.patch
          39 kB
          Íñigo Goiri
        4. YARN-3534-3.patch
          39 kB
          Íñigo Goiri
        5. YARN-3534-4.patch
          9 kB
          Íñigo Goiri
        6. YARN-3534-5.patch
          29 kB
          Íñigo Goiri
        7. YARN-3534-6.patch
          9 kB
          Íñigo Goiri
        8. YARN-3534-7.patch
          9 kB
          Íñigo Goiri
        9. YARN-3534-8.patch
          9 kB
          Íñigo Goiri
        10. YARN-3534-9.patch
          8 kB
          Íñigo Goiri
        11. YARN-3534-10.patch
          15 kB
          Íñigo Goiri
        12. YARN-3534-11.patch
          17 kB
          Íñigo Goiri
        13. YARN-3534-12.patch
          18 kB
          Íñigo Goiri
        14. YARN-3534-14.patch
          11 kB
          Íñigo Goiri
        15. YARN-3534-15.patch
          12 kB
          Íñigo Goiri
        16. YARN-3534-16.patch
          15 kB
          Íñigo Goiri
        17. YARN-3534-16.patch
          15 kB
          Íñigo Goiri
        18. YARN-3534-17.patch
          15 kB
          Íñigo Goiri
        19. YARN-3534-17.patch
          15 kB
          Íñigo Goiri
        20. YARN-3534-18.patch
          16 kB
          Íñigo Goiri

        Issue Links

        Activity

          sunilg Sunil G added a comment -

          resource utilization of the nodes

          Is this going to be configurable item to look for, or all resources will be monitored?

          sunilg Sunil G added a comment - resource utilization of the nodes Is this going to be configurable item to look for, or all resources will be monitored?
          elgoiri Íñigo Goiri added a comment -

          Sunil G, my plan is to send the same entity as in YARN-3482: ResourceUtilization. For now, it will have memory in MB and Virtual Cores as a float. We can extend this for other resources.
          In terms of the collection of the utilization, this will be monitored by the NodeResourceMonitor (currently empty) which will mimic the behavior of the ContainerMonitor which uses two configuration parameters: (1) if we monitor the resources of the node and (2) how often the resources are monitored.

          I'm in the last stages of testing it and tomorrow I'll post a patch with this change. The structure was already there so it's a matter of adding the communication part..

          elgoiri Íñigo Goiri added a comment - Sunil G , my plan is to send the same entity as in YARN-3482 : ResourceUtilization. For now, it will have memory in MB and Virtual Cores as a float. We can extend this for other resources. In terms of the collection of the utilization, this will be monitored by the NodeResourceMonitor (currently empty) which will mimic the behavior of the ContainerMonitor which uses two configuration parameters: (1) if we monitor the resources of the node and (2) how often the resources are monitored. I'm in the last stages of testing it and tomorrow I'll post a patch with this change. The structure was already there so it's a matter of adding the communication part..
          elgoiri Íñigo Goiri added a comment -

          This does not include unit tests yet. Working on them.

          elgoiri Íñigo Goiri added a comment - This does not include unit tests yet. Working on them.
          elgoiri Íñigo Goiri added a comment -

          No unit tests yet.

          elgoiri Íñigo Goiri added a comment - No unit tests yet.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727762/YARN-3534-1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0b3f895
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7484/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727762/YARN-3534-1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0b3f895 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7484/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Merged to trunk.

          elgoiri Íñigo Goiri added a comment - Merged to trunk.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          -1 whitespace 0m 0s The patch has 78 line(s) that end in whitespace.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 5m 32s The applied patch generated 11 additional checkstyle issues.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 5m 55s Tests passed in hadoop-yarn-server-nodemanager.
              53m 2s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 whitespace 0m 0s The patch has 78 line(s) that end in whitespace. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 32s The applied patch generated 11 additional checkstyle issues. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 5m 55s Tests passed in hadoop-yarn-server-nodemanager.     53m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727766/YARN-3534-2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0b3f895 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/whitespace.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7485/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7485/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7485/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Removing spaces at the end of the line; Jenkins doesn't seem to like them anymore.

          elgoiri Íñigo Goiri added a comment - Removing spaces at the end of the line; Jenkins doesn't seem to like them anymore.
          elgoiri Íñigo Goiri added a comment -

          Updating trunk before it goes ahead.

          elgoiri Íñigo Goiri added a comment - Updating trunk before it goes ahead.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 38s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 29s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 7m 47s The applied patch generated 11 additional checkstyle issues.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.
              55m 33s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 38s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 7m 47s The applied patch generated 11 additional checkstyle issues. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 37s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 1s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 5s Tests passed in hadoop-yarn-server-nodemanager.     55m 33s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727786/YARN-3534-3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcf89dd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7489/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7489/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7489/console This message was automatically generated.

          Skimmed through the latest patch. High-level comments/questions:

          1. Do we need a separate class/proto for ResourceUtilization? Could we just reuse Resource? That should make the patch significantly small.
          2. Would be nice to have NodeResourceMonitor emit metrics for usage. We could do this on a follow-up JIRA.
          kasha Karthik Kambatla added a comment - Skimmed through the latest patch. High-level comments/questions: Do we need a separate class/proto for ResourceUtilization? Could we just reuse Resource? That should make the patch significantly small. Would be nice to have NodeResourceMonitor emit metrics for usage. We could do this on a follow-up JIRA.

          I notice the patch tries to provide utilization information in bytes for memory and float for CPU. Since the RM schedules in MB and vcores, seeing the utilization as rounded-up values in a Resource object is probably enough.

          kasha Karthik Kambatla added a comment - I notice the patch tries to provide utilization information in bytes for memory and float for CPU. Since the RM schedules in MB and vcores, seeing the utilization as rounded-up values in a Resource object is probably enough.
          elgoiri Íñigo Goiri added a comment -

          The original reason for having ResourceUtilization was getting better granularity for the CPUs. We had some discussion about it in YARN-3481; take a look there and chim in.

          My original implementation had Resource but when trying to do scheduling of containers based on this, there were a lot of holes in the scheduling. Given this, I thought this patch was a good place to create this new utilization entity with CPU as a float.

          Regarding the metrics in the NodeResourceMonitor, I completely agree. I thought about doing it right away but as you mentioned, it seemed a better idea to save it for another JIRA. Let's do that.

          elgoiri Íñigo Goiri added a comment - The original reason for having ResourceUtilization was getting better granularity for the CPUs. We had some discussion about it in YARN-3481 ; take a look there and chim in. My original implementation had Resource but when trying to do scheduling of containers based on this, there were a lot of holes in the scheduling. Given this, I thought this patch was a good place to create this new utilization entity with CPU as a float. Regarding the metrics in the NodeResourceMonitor, I completely agree. I thought about doing it right away but as you mentioned, it seemed a better idea to save it for another JIRA. Let's do that.

          Thanks for pointing me to YARN-3481, Inigo. Didn't see the other JIRA.

          Reading through those comments, I agree it might be better to add a separate class/proto for ResourceUtilization. Efforts like Tetris could add disk/network utilization even if the API itself doesn't support them explicitly.

          kasha Karthik Kambatla added a comment - Thanks for pointing me to YARN-3481 , Inigo. Didn't see the other JIRA. Reading through those comments, I agree it might be better to add a separate class/proto for ResourceUtilization. Efforts like Tetris could add disk/network utilization even if the API itself doesn't support them explicitly.

          Please see my comment on YARN-3481.

          If nothing, we should separate the reporting bits from the measurement bit?

          Also, we don't need the ResourceUtilizationCalculator layers? Once the information reaches the ResourceManager, RM can use the regular Resources utils.

          vinodkv Vinod Kumar Vavilapalli added a comment - Please see my comment on YARN-3481 . If nothing, we should separate the reporting bits from the measurement bit? Also, we don't need the ResourceUtilizationCalculator layers? Once the information reaches the ResourceManager, RM can use the regular Resources utils.
          elgoiri Íñigo Goiri added a comment -

          I could first do the NodeResourceMonitorImpl and then add that info it to the heartbeat in a second JIRA. However, I thought the addition to the heartbeat was simple enough. Anyways, up to you; I can split it in two.

          Regarding the extra ResourceUtilizations layers, I could get rid of them for now as it's not currently needed. However, in parallel, I'm privately working on a scheduler which is aware of utilization and it's much better to use those layers as I need the floats for the CPUs instead of plain int. I thought about making ResourceUtilizaiton an extension of Resource but it does not work...

          Anyway, I'll send a patch without the extra ResourceUtilizations layer as it's not really required for this patch. Does that sound reasonable?

          elgoiri Íñigo Goiri added a comment - I could first do the NodeResourceMonitorImpl and then add that info it to the heartbeat in a second JIRA. However, I thought the addition to the heartbeat was simple enough. Anyways, up to you; I can split it in two. Regarding the extra ResourceUtilizations layers, I could get rid of them for now as it's not currently needed. However, in parallel, I'm privately working on a scheduler which is aware of utilization and it's much better to use those layers as I need the floats for the CPUs instead of plain int. I thought about making ResourceUtilizaiton an extension of Resource but it does not work... Anyway, I'll send a patch without the extra ResourceUtilizations layer as it's not really required for this patch. Does that sound reasonable?
          vvasudev Varun Vasudev added a comment -

          Thanks for the path Íñigo Goiri! Is it possible for you to split up the patch into two - one to record the memory and cpu utilization in NodeResourceMonitorImpl and one for the RPC changes. If you could file a separate JIRA for the NodeResourceMonitorImpl changes that you have made(they are definitely required and should go in), that would be ideal.

          I'm not sure if we should use RPC to expose the stats. Using RPC restricts the stats to a limited number of services whereas using REST lets a larger number of services access the stats - which is why YARN-3332 wants to use REST to expose the NM stats. Using REST should also make it easier to add more stats with respect to utilization.

          vvasudev Varun Vasudev added a comment - Thanks for the path Íñigo Goiri ! Is it possible for you to split up the patch into two - one to record the memory and cpu utilization in NodeResourceMonitorImpl and one for the RPC changes. If you could file a separate JIRA for the NodeResourceMonitorImpl changes that you have made(they are definitely required and should go in), that would be ideal. I'm not sure if we should use RPC to expose the stats. Using RPC restricts the stats to a limited number of services whereas using REST lets a larger number of services access the stats - which is why YARN-3332 wants to use REST to expose the NM stats. Using REST should also make it easier to add more stats with respect to utilization.

          I'm not sure if we should use RPC to expose the stats.

          As YARN-3332 suggests, we should expose these stats through REST. However, for the RM (scheduler) to get this information, I think RPC (through node heartbeat) is better. That way, the RM doesn't have to poll the REST API for every node there is an NM heartbeat for.

          If the suggestion is for the NM to get this information from the REST API, I think that is fine. However, since this communication is private to NM, we can always change this later once YARN-3332 is ready.

          kasha Karthik Kambatla added a comment - I'm not sure if we should use RPC to expose the stats. As YARN-3332 suggests, we should expose these stats through REST. However, for the RM (scheduler) to get this information, I think RPC (through node heartbeat) is better. That way, the RM doesn't have to poll the REST API for every node there is an NM heartbeat for. If the suggestion is for the NM to get this information from the REST API, I think that is fine. However, since this communication is private to NM, we can always change this later once YARN-3332 is ready.
          elgoiri Íñigo Goiri added a comment -

          It looks like there's a consensus on splitting on monitoring and reporting. If everybody is cool with it, I will rename this JIRA to "Collect node resource utilization" and just do the implementaiton of NodeResourceMonitor.

          The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?

          elgoiri Íñigo Goiri added a comment - It looks like there's a consensus on splitting on monitoring and reporting. If everybody is cool with it, I will rename this JIRA to "Collect node resource utilization" and just do the implementaiton of NodeResourceMonitor. The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332 )?
          elgoiri Íñigo Goiri added a comment -

          Isolating the changes for the NodeResourceMonitorImpl. The aggregationg of data and sending to the RM will be done in another JIRA.

          elgoiri Íñigo Goiri added a comment - Isolating the changes for the NodeResourceMonitorImpl. The aggregationg of data and sending to the RM will be done in another JIRA.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          -1 javac 1m 48s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729617/YARN-3534-4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f0db797
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7560/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. -1 javac 1m 48s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729617/YARN-3534-4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f0db797 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7560/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Fixed compilation issue with the interface.

          elgoiri Íñigo Goiri added a comment - Fixed compilation issue with the interface.
          hadoopqa Hadoop QA added a comment -

          A patch to test-patch or smart-apply-patch has been detected.
          Re-executing against the patched versions to perform further tests.
          The console is at https://builds.apache.org/job/PreCommit-YARN-Build/7564/console in case of problems.

          hadoopqa Hadoop QA added a comment - A patch to test-patch or smart-apply-patch has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/7564/console in case of problems.

          Converting this to be a sub-task of YARN-1011. See my last comment at YARN-3481.

          vinodkv Vinod Kumar Vavilapalli added a comment - Converting this to be a sub-task of YARN-1011 . See my last comment at YARN-3481 .
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s dev-support patch detected.
          0 pre-patch 14m 45s Pre-patch trunk compilation is healthy.
          0 @author 0m 0s Skipping @author checks as test-patch has been patched.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 6m 29s There were no new javac warning messages.
          +1 javadoc 7m 25s There were no new javadoc warning messages.
          -1 release audit 22m 41s The applied patch generated 364 release audit warnings.
          -1 checkstyle 4m 26s The applied patch generated 2 additional checkstyle issues.
          0 shellcheck 4m 26s Shellcheck was not available.
          +1 install 1m 12s mvn install still works.
          +1 eclipse:eclipse 0m 30s The patch built with eclipse:eclipse.
          +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 common tests 23m 7s Tests passed in hadoop-common.
          +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api.
          -1 yarn tests 5m 47s Tests failed in hadoop-yarn-server-nodemanager.
              90m 58s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s dev-support patch detected. 0 pre-patch 14m 45s Pre-patch trunk compilation is healthy. 0 @author 0m 0s Skipping @author checks as test-patch has been patched. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 6m 29s There were no new javac warning messages. +1 javadoc 7m 25s There were no new javadoc warning messages. -1 release audit 22m 41s The applied patch generated 364 release audit warnings. -1 checkstyle 4m 26s The applied patch generated 2 additional checkstyle issues. 0 shellcheck 4m 26s Shellcheck was not available. +1 install 1m 12s mvn install still works. +1 eclipse:eclipse 0m 30s The patch built with eclipse:eclipse. +1 findbugs 4m 4s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 23m 7s Tests passed in hadoop-common. +1 yarn tests 0m 29s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 47s Tests failed in hadoop-yarn-server-nodemanager.     90m 58s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729676/YARN-3534-5.patch Optional Tests shellcheck javadoc javac unit findbugs checkstyle git revision trunk / 98a6176 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/checkstyle-result-diff.txt hadoop-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7564/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7564/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7564/console This message was automatically generated.
          vvasudev Varun Vasudev added a comment -

          Íñigo Goiri - the latest patch(YARN-3534-5.patch) has modifications to the checkstyle script as well as OsSecureRandom.java. I assume those changes are not meant to be part of your patch. Can you please upload a patch with just your changes? Thanks!

          vvasudev Varun Vasudev added a comment - Íñigo Goiri - the latest patch( YARN-3534 -5.patch) has modifications to the checkstyle script as well as OsSecureRandom.java. I assume those changes are not meant to be part of your patch. Can you please upload a patch with just your changes? Thanks!
          elgoiri Íñigo Goiri added a comment -

          Diff to the latest trunk.

          elgoiri Íñigo Goiri added a comment - Diff to the latest trunk.
          elgoiri Íñigo Goiri added a comment -

          Sorry about that, I did the diff and then I did the patch 10 minutes after... meanwhile there were change sin trunk and it got all messed up. I think this one should be OK. No unit tests though.

          elgoiri Íñigo Goiri added a comment - Sorry about that, I did the diff and then I did the patch 10 minutes after... meanwhile there were change sin trunk and it got all messed up. I think this one should be OK. No unit tests though.
          hadoopqa Hadoop QA added a comment -

          The patch artifact directory has been removed!
          This is a fatal error for test-patch.sh. Aborting.
          Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7569/ may provide some hints.

          hadoopqa Hadoop QA added a comment - The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H4) information at https://builds.apache.org/job/PreCommit-YARN-Build/7569/ may provide some hints.
          elgoiri Íñigo Goiri added a comment -

          Trying to trigger built from latest trunk

          elgoiri Íñigo Goiri added a comment - Trying to trigger built from latest trunk
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 javadoc 9m 36s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 211, now 213).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.
              44m 33s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729860/YARN-3534-7.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 6f541ed
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7589/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7589/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 33s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 22s The applied patch generated 2 new checkstyle issues (total was 211, now 213). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 24s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.     44m 33s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729860/YARN-3534-7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6f541ed checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7589/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7589/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7589/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Fixing code style issues (nwo the output of the checker is meaningful so life is much easier).

          The broken tests weren't related to my changes.

          Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.

          elgoiri Íñigo Goiri added a comment - Fixing code style issues (nwo the output of the checker is meaningful so life is much easier). The broken tests weren't related to my changes. Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 39s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 30s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 28s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api.
          +1 yarn tests 5m 51s Tests passed in hadoop-yarn-server-nodemanager.
              44m 26s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12729911/YARN-3534-8.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7594/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7594/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 39s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 30s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 28s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 23s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 30s Tests passed in hadoop-yarn-api. +1 yarn tests 5m 51s Tests passed in hadoop-yarn-server-nodemanager.     44m 26s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12729911/YARN-3534-8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7594/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7594/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7594/console This message was automatically generated.

          The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332)?

          I believe there are existing tickets for this. I started consolidating all the related efforts, let me do that so that you can start working on this once I identify the right JIRA.

          Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply.

          You can reuse some of the code at TestLinuxResourceCalculatorPlugin and write a new TestNodeResourceMonitor that measures utilization against mock information.

          Couple other comments:

          • context variable is unused
          • Can we also capture vmem usage as well as cpu-wall time?
          • As per my proposal at YARN-3332, I'd like to refactor this later to be the statistics-agent. So, for now, can you just move the configuration-properties from YarnConfiguration into NodeResourceMonitorImpl?
          vinodkv Vinod Kumar Vavilapalli added a comment - The argument of how to pass this information to the RM seems still open. Should I open a new JIRA for that and move the discussion about using REST, heartbeat etc to that one? Or should we have that discussion in one of the related JIRAs (e.g., YARN-3332 )? I believe there are existing tickets for this. I started consolidating all the related efforts, let me do that so that you can start working on this once I identify the right JIRA. Any proposal for unit tests? The ones in ContainerMonitorImpl don't really apply. You can reuse some of the code at TestLinuxResourceCalculatorPlugin and write a new TestNodeResourceMonitor that measures utilization against mock information. Couple other comments: context variable is unused Can we also capture vmem usage as well as cpu-wall time? As per my proposal at YARN-3332 , I'd like to refactor this later to be the statistics-agent. So, for now, can you just move the configuration-properties from YarnConfiguration into NodeResourceMonitorImpl?
          elgoiri Íñigo Goiri added a comment -

          Removed nodemanager context.
          Added vmem (I cannot put it inside the Resource though).

          elgoiri Íñigo Goiri added a comment - Removed nodemanager context. Added vmem (I cannot put it inside the Resource though).
          elgoiri Íñigo Goiri added a comment -

          Where do you want to put the vmem and where to get the cpu-wall time?
          Regarding YarnConfiguration, I didn't understand what you wanted to do there. Do you want to put the constants in the class itself?

          elgoiri Íñigo Goiri added a comment - Where do you want to put the vmem and where to get the cpu-wall time? Regarding YarnConfiguration, I didn't understand what you wanted to do there. Do you want to put the constants in the class itself?

          Thanks for working on this, Inigo.

          Few comments:

          1. Given these stats are to be sent to the RM in heartbeat, should we capture the aggregate node resource usage in ResourceUtilization?
          2. Instead of adding a separate DEFAULT_NM_NODE_MON_INTERVAL_MS, we should probably just re-use the default for container-monitor?
          3. Also, should we add another config yarn.nodemanager.usage-monitor.interval-ms that both container-monitor and node-monitor inherit unless specified otherwise? If that seems reasonable, we should deprecate the default value for container-monitor-interval.
          4. For the monitoring thread,
            1. set the thread name?
            2. make it a daemon thread?
            3. on monitoringThread.join(), specify a timeout as well.
            4. in the corresponding catch-block, at least log that we couldn't wait until the monitoring-thread is interrupted.
          kasha Karthik Kambatla added a comment - Thanks for working on this, Inigo. Few comments: Given these stats are to be sent to the RM in heartbeat, should we capture the aggregate node resource usage in ResourceUtilization ? Instead of adding a separate DEFAULT_NM_NODE_MON_INTERVAL_MS , we should probably just re-use the default for container-monitor? Also, should we add another config yarn.nodemanager.usage-monitor.interval-ms that both container-monitor and node-monitor inherit unless specified otherwise? If that seems reasonable, we should deprecate the default value for container-monitor-interval. For the monitoring thread, set the thread name? make it a daemon thread? on monitoringThread.join() , specify a timeout as well. in the corresponding catch-block, at least log that we couldn't wait until the monitoring-thread is interrupted.
          elgoiri Íñigo Goiri added a comment -

          I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here.

          elgoiri Íñigo Goiri added a comment - I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here.
          elgoiri Íñigo Goiri added a comment -

          Solved some comments

          elgoiri Íñigo Goiri added a comment - Solved some comments
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 16m 42s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 34s The applied patch generated 1 additional warning messages.
          -1 javadoc 9m 39s The applied patch generated 3 additional warning messages.
          -1 release audit 0m 18s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 24s The applied patch generated 9 new checkstyle issues (total was 212, now 220).
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.
              50m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12737047/YARN-3534-10.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c59e745
          javac https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavacWarnings.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavadocWarnings.txt
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/patchReleaseAuditProblems.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/whitespace.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8176/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8176/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 16m 42s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 34s The applied patch generated 1 additional warning messages. -1 javadoc 9m 39s The applied patch generated 3 additional warning messages. -1 release audit 0m 18s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 24s The applied patch generated 9 new checkstyle issues (total was 212, now 220). -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 18s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 4s Tests passed in hadoop-yarn-server-nodemanager.     50m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737047/YARN-3534-10.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 javac https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavacWarnings.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavadocWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/whitespace.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8176/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8176/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Fixing build issues

          elgoiri Íñigo Goiri added a comment - Fixing build issues
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 18s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          -1 release audit 0m 18s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 212, now 211).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 13s Tests passed in hadoop-yarn-server-nodemanager.
              52m 52s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 18s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. -1 release audit 0m 18s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 44s The applied patch generated 1 new checkstyle issues (total was 212, now 211). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 21s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 13s Tests passed in hadoop-yarn-server-nodemanager.     52m 52s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12737322/YARN-3534-11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c59e745 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8180/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8180/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Added license for ResourceUtilizations and extra comments.

          elgoiri Íñigo Goiri added a comment - Added license for ResourceUtilizations and extra comments.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 30s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 213, now 212).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.
              53m 3s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 30s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. -1 checkstyle 1m 41s The applied patch generated 1 new checkstyle issues (total was 213, now 212). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.     53m 3s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739505/YARN-3534-12.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b8341f1 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8247/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8247/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Updated to trunk (using ResourceUtilization already there).

          elgoiri Íñigo Goiri added a comment - Updated to trunk (using ResourceUtilization already there).
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 2m 59s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ac60483
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8483/console

          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 2m 59s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ac60483 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8483/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Fixed issue with constants.
          We need to agree that these names are OK and that we are fine with deprecating the old ones.

          elgoiri Íñigo Goiri added a comment - Fixed issue with constants. We need to agree that these names are OK and that we are fine with deprecating the old ones.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 7m 46s The applied patch generated 2 additional warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 19s The applied patch generated 5 new checkstyle issues (total was 211, now 215).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.
              48m 0s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
            hadoop.yarn.server.nodemanager.containermanager.container.TestContainer
            hadoop.yarn.server.nodemanager.TestDeletionService



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 7m 46s The applied patch generated 2 additional warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 19s The applied patch generated 5 new checkstyle issues (total was 211, now 215). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. -1 yarn tests 6m 6s Tests failed in hadoop-yarn-server-nodemanager.     48m 0s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService   hadoop.yarn.server.nodemanager.containermanager.container.TestContainer   hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12744609/YARN-3534-15.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 1a0752d javac https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffJavacWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8488/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8488/console This message was automatically generated.

          Thanks for updating the patch, Íñigo Goiri. The latest patch looks pretty good. Few minor comments:

          1. Let us not deprecate NM_CONTAINER_MON_INTERVAL_MS. Deprecating the default value seems reasonable. One might want to specify a different interval for node monitoring and container monitoring.
          2. Make corresponding changes to yarn-site.xml
          3. NodeResourceMonitorImpl#serviceStop: the log message should say couldn't wait for the thread to join, and may be we should log it as WARN instead of INFO
          4. ContainersMonitorImpl: Users could continue to use the deprecated configs. If those are specified and the new ones are not, we should fallback to the deprecated ones. If this becomes too complicated, I am comfortable with leaving the configs for container-monitoring as is. We could file a follow-up JIRA to clean these configs for Hadoop-3.x.
          kasha Karthik Kambatla added a comment - Thanks for updating the patch, Íñigo Goiri . The latest patch looks pretty good. Few minor comments: Let us not deprecate NM_CONTAINER_MON_INTERVAL_MS . Deprecating the default value seems reasonable. One might want to specify a different interval for node monitoring and container monitoring. Make corresponding changes to yarn-site.xml NodeResourceMonitorImpl#serviceStop : the log message should say couldn't wait for the thread to join, and may be we should log it as WARN instead of INFO ContainersMonitorImpl: Users could continue to use the deprecated configs. If those are specified and the new ones are not, we should fallback to the deprecated ones. If this becomes too complicated, I am comfortable with leaving the configs for container-monitoring as is. We could file a follow-up JIRA to clean these configs for Hadoop-3.x.
          elgoiri Íñigo Goiri added a comment -

          Solved most of Karthik Kambatla's comments.

          elgoiri Íñigo Goiri added a comment - Solved most of Karthik Kambatla 's comments.
          elgoiri Íñigo Goiri added a comment -

          I guess in #2 you meant yarn-default.xml, right?
          I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want.

          elgoiri Íñigo Goiri added a comment - I guess in #2 you meant yarn-default.xml, right? I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 46s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 211, now 214).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.
              53m 43s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 46s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 4 new checkstyle issues (total was 211, now 214). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 9s Tests passed in hadoop-yarn-server-nodemanager.     53m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748272/YARN-3534-16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d311a38 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8742/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8742/console This message was automatically generated.
          elgoiri Íñigo Goiri added a comment -

          Fixing checkstyle

          elgoiri Íñigo Goiri added a comment - Fixing checkstyle
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 2m 33s The applied patch generated 2 new checkstyle issues (total was 28, now 27).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.
              53m 36s  



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 48s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 2m 33s The applied patch generated 2 new checkstyle issues (total was 28, now 27). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.     53m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748290/YARN-3534-16.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d311a38 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8747/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8747/console This message was automatically generated.

          Hopefully, last set of comments:

          1. We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should have been more explicit in my previous comment.
          2. ContainersMonitorImpl should prioritize container-specific configs and fallback if they are not present. Something along these lines:
                this.monitoringInterval =
                    conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS,
                        conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS,
                            YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS));
            
                Class<? extends ResourceCalculatorPlugin> clazz =
                    conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR,
                        conf.getClass(
                            YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null,
                            ResourceCalculatorPlugin.class),
                        ResourceCalculatorPlugin.class);
            
          3. yarn-default.xml needs to be updated for the CONTAINER_MON configs we are changing the behavior of, and any default values there should be removed.
          kasha Karthik Kambatla added a comment - Hopefully, last set of comments: We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should have been more explicit in my previous comment. ContainersMonitorImpl should prioritize container-specific configs and fallback if they are not present. Something along these lines: this .monitoringInterval = conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS, conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS, YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS)); Class <? extends ResourceCalculatorPlugin> clazz = conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR, conf.getClass( YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null , ResourceCalculatorPlugin.class), ResourceCalculatorPlugin.class); yarn-default.xml needs to be updated for the CONTAINER_MON configs we are changing the behavior of, and any default values there should be removed.
          elgoiri Íñigo Goiri added a comment -
          1. I missed the Deprecated for the calculator... my bad. Fixed.
          2. That's good; fixed.
          3. The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though.
          elgoiri Íñigo Goiri added a comment - I missed the Deprecated for the calculator... my bad. Fixed. That's good; fixed. The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though.
          elgoiri Íñigo Goiri added a comment -

          Incorporated Karthik Kambatla improvements but still not clear on the yarn-default.xml.

          elgoiri Íñigo Goiri added a comment - Incorporated Karthik Kambatla improvements but still not clear on the yarn-default.xml.

          In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used."

          kasha Karthik Kambatla added a comment - In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used."
          elgoiri Íñigo Goiri added a comment -

          Fixed yarn-default.xml description.

          elgoiri Íñigo Goiri added a comment - Fixed yarn-default.xml description.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 45s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 3s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common.
          -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.
              53m 35s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver
            hadoop.yarn.server.nodemanager.TestDeletionService



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 45s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 45s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 3s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 22s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common. -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.     53m 35s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749432/YARN-3534-17.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8801/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8801/console This message was automatically generated.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
          +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.
              53m 54s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 51s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 20s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.     53m 54s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749435/YARN-3534-17.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8802/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8802/console This message was automatically generated.

          The latest patches (v17) don't seem to have the updates. Íñigo Goiri - could you please verify?

          kasha Karthik Kambatla added a comment - The latest patches (v17) don't seem to have the updates. Íñigo Goiri - could you please verify?
          elgoiri Íñigo Goiri added a comment -

          Fixed xml defaults.

          elgoiri Íñigo Goiri added a comment - Fixed xml defaults.
          elgoiri Íñigo Goiri added a comment -

          I uploaded 17 twice instead of 18... sorry.

          elgoiri Íñigo Goiri added a comment - I uploaded 17 twice instead of 18... sorry.
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common.
          +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.
              53m 33s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver



          This message was automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 46s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 4m 21s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 55s Tests failed in hadoop-yarn-common. +1 yarn tests 6m 10s Tests passed in hadoop-yarn-server-nodemanager.     53m 33s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12750666/YARN-3534-18.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8dfec7a checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8851/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8851/console This message was automatically generated.

          +1

          kasha Karthik Kambatla added a comment - +1

          Just committed to trunk and branch-2.

          Thanks Íñigo Goiri for the contribution and patience through the reviews.

          kasha Karthik Kambatla added a comment - Just committed to trunk and branch-2. Thanks Íñigo Goiri for the contribution and patience through the reviews.
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8311/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8311/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          elgoiri Íñigo Goiri added a comment -

          Thank you Karthik Kambatla for taking care of the review and the commits.
          I'll be moving to propagating this info to the scheduler.

          elgoiri Íñigo Goiri added a comment - Thank you Karthik Kambatla for taking care of the review and the commits. I'll be moving to propagating this info to the scheduler.
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/)
          YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/ ) YARN-3534 . Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java

          Íñigo Goiri, Karthik Kambatla, could you comment on extending this to also take in network and disk usage information?

          srikanthkandula Srikanth Kandula added a comment - Íñigo Goiri , Karthik Kambatla , could you comment on extending this to also take in network and disk usage information?

          People

            elgoiri Íñigo Goiri
            elgoiri Íñigo Goiri
            Votes:
            0 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 336h
                336h
                Remaining:
                Remaining Estimate - 336h
                336h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Slack

                  Issue deployment

                    Client must be authenticated to access this resource.