Hadoop YARN
  1. Hadoop YARN
  2. YARN-160

nodemanagers should obtain cpu/memory values from underlying OS

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.8.0
    • Component/s: nodemanager
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      As mentioned in YARN-2

      NM memory and CPU configs

      Currently these values are coming from the config of the NM, we should be able to obtain those values from the OS (ie, in the case of Linux from /proc/meminfo & /proc/cpuinfo). As this is highly OS dependent we should have an interface that obtains this information. In addition implementations of this interface should be able to specify a mem/cpu offset (amount of mem/cpu not to be avail as YARN resource), this would allow to reserve mem/cpu for the OS and other services outside of YARN containers.

      1. YARN-160.008.patch
        56 kB
        Varun Vasudev
      2. YARN-160.007.patch
        56 kB
        Varun Vasudev
      3. YARN-160.006.patch
        55 kB
        Varun Vasudev
      4. YARN-160.005.patch
        55 kB
        Varun Vasudev
      5. apache-yarn-160.3.patch
        39 kB
        Varun Vasudev
      6. apache-yarn-160.2.patch
        38 kB
        Varun Vasudev
      7. apache-yarn-160.1.patch
        24 kB
        Varun Vasudev
      8. apache-yarn-160.0.patch
        24 kB
        Varun Vasudev

        Issue Links

          Activity

          Hide
          Radim Kolar added a comment -

          there is a interface from obtaining these values from OS. Its called resourcecalculator plugin. It might be good to do autodetection but i need ability to override these values in config.

          Show
          Radim Kolar added a comment - there is a interface from obtaining these values from OS. Its called resourcecalculator plugin. It might be good to do autodetection but i need ability to override these values in config.
          Hide
          Alejandro Abdelnur added a comment -

          Radim, thanks for pointing that. This it seems it is not wired for the NMs to report such info back to the RM, looking at the code NodeManager creates a NodeStatusUpdater using values from configuration files. Only the ContainersMonitorIMpl reads the underlying memory. So this JIRA would be to get the NodeStatusUpdater to use a resourcecalculator. Makes sense?

          Show
          Alejandro Abdelnur added a comment - Radim, thanks for pointing that. This it seems it is not wired for the NMs to report such info back to the RM, looking at the code NodeManager creates a NodeStatusUpdater using values from configuration files. Only the ContainersMonitorIMpl reads the underlying memory. So this JIRA would be to get the NodeStatusUpdater to use a resourcecalculator. Makes sense?
          Hide
          Radim Kolar added a comment -

          My proposal is to make entire new framework for resource management - MAPREDUCE-4256

          Show
          Radim Kolar added a comment - My proposal is to make entire new framework for resource management - MAPREDUCE-4256
          Hide
          Timothy St. Clair added a comment -

          If it's possible to tag along development on this one, I would be interested in the approach. IMHO referencing existing solutions gauges baseline:

          Ref:
          http://www.open-mpi.org/projects/hwloc/
          http://www.rce-cast.com/Podcast/rce-33-hwloc-portable-hardware-locality.html
          http://gridscheduler.sourceforge.net/projects/hwloc/GridEnginehwloc.html

          Show
          Timothy St. Clair added a comment - If it's possible to tag along development on this one, I would be interested in the approach. IMHO referencing existing solutions gauges baseline: Ref: http://www.open-mpi.org/projects/hwloc/ http://www.rce-cast.com/Podcast/rce-33-hwloc-portable-hardware-locality.html http://gridscheduler.sourceforge.net/projects/hwloc/GridEnginehwloc.html
          Hide
          Arun C Murthy added a comment -

          I'd like to push to get this done asap.

          [~t.st.clair] thanks for pointer to hwloc. In your opinion, should we directly use hwloc in YARN rather than invent our own? What has been your experience with hwloc? It does have an appropriate license (BSD - http://www.open-mpi.org/projects/hwloc/license.php) ...

          Show
          Arun C Murthy added a comment - I'd like to push to get this done asap. [~t.st.clair] thanks for pointer to hwloc. In your opinion, should we directly use hwloc in YARN rather than invent our own? What has been your experience with hwloc? It does have an appropriate license (BSD - http://www.open-mpi.org/projects/hwloc/license.php ) ...
          Hide
          Arun C Murthy added a comment -

          I'm particularly interested in getting this done in light of YARN-1024 - we should look for something we can normalize down to YVC (or ECU).

          Show
          Arun C Murthy added a comment - I'm particularly interested in getting this done in light of YARN-1024 - we should look for something we can normalize down to YVC (or ECU).
          Hide
          Arun C Murthy added a comment -

          Alejandro Abdelnur Is there a chance you plan on working on this? I'd like to get this into 2.3.0 if possible. Thanks!

          Show
          Arun C Murthy added a comment - Alejandro Abdelnur Is there a chance you plan on working on this? I'd like to get this into 2.3.0 if possible. Thanks!
          Hide
          Timothy St. Clair added a comment -

          I think the prudent approach would be to evaluate hwloc and its community, and determine if it meets the internal needs of YARN. For risk mitigation purposes, I think having a plugin abstraction layer as a fallback would also be wise.

          I did notice there are also java bindings around hwloc (https://launchpad.net/jhwloc/)

          Show
          Timothy St. Clair added a comment - I think the prudent approach would be to evaluate hwloc and its community, and determine if it meets the internal needs of YARN. For risk mitigation purposes, I think having a plugin abstraction layer as a fallback would also be wise. I did notice there are also java bindings around hwloc ( https://launchpad.net/jhwloc/ )
          Hide
          Alejandro Abdelnur added a comment -

          I have my hands full at the moment, I won't be able to take onto this one for a while.

          Making it unassigned in case somebody wants to take a stab to it.

          Show
          Alejandro Abdelnur added a comment - I have my hands full at the moment, I won't be able to take onto this one for a while. Making it unassigned in case somebody wants to take a stab to it.
          Hide
          Varun Vasudev added a comment -

          Patch to automatically calculate cpu and memory from the OS. In case of cpu, I've added a flag to allow admins to decide if they want to count hyperthreads as vcores or not(by default they are counted as vcores). The flag currently works on Linux only.

          In case of memory, I calculate the memory for containers as 80% of (RAM - 2*HADOOP_HEAPSIZE), to account for memory used by the datanode and the nodemanager.

          I've also changed the default behaviour to use the calculated values instead of 8 and 8192 for cpu and memory that we have been using till now.

          Feedback would be welcome.

          Show
          Varun Vasudev added a comment - Patch to automatically calculate cpu and memory from the OS. In case of cpu, I've added a flag to allow admins to decide if they want to count hyperthreads as vcores or not(by default they are counted as vcores). The flag currently works on Linux only. In case of memory, I calculate the memory for containers as 80% of (RAM - 2*HADOOP_HEAPSIZE), to account for memory used by the datanode and the nodemanager. I've also changed the default behaviour to use the calculated values instead of 8 and 8192 for cpu and memory that we have been using till now. Feedback would be welcome.
          Hide
          Junping Du added a comment -

          Thanks Varun Vasudev for working on this. Just take a quick glance, a few comments:

          • The old way to configure resource of NM is still useful, especially when there are other agents running (like: HBase RegionServer). Thus, user need flexibility to calculate resource themselves in some cases, so we should provide another new option instead of removing old way completely.
          • Given this is a new feature, we shouldn't change cluster's behavior with old configuration in upgrade prospective. We should keep previous configuration work as usual especially when user use some default settings.
          Show
          Junping Du added a comment - Thanks Varun Vasudev for working on this. Just take a quick glance, a few comments: The old way to configure resource of NM is still useful, especially when there are other agents running (like: HBase RegionServer). Thus, user need flexibility to calculate resource themselves in some cases, so we should provide another new option instead of removing old way completely. Given this is a new feature, we shouldn't change cluster's behavior with old configuration in upgrade prospective. We should keep previous configuration work as usual especially when user use some default settings.
          Hide
          Varun Vasudev added a comment -

          Junping Du

          The old way to configure resource of NM is still useful, especially when there are other agents running (like: HBase RegionServer). Thus, user need flexibility to calculate resource themselves in some cases, so we should provide another new option instead of removing old way completely.

          The patch supports the old way. If a user has set values for memory and vcores, they're used without looking at the underlying hardware. I've added test cases to verify that behaviour as well. Have I missed a use case?

          Given this is a new feature, we shouldn't change cluster's behavior with old configuration in upgrade prospective. We should keep previous configuration work as usual especially when user use some default settings.

          There are two scenarios here -
          1. A configuration file with custom settings for memory and cpu - nothing will change for these users.
          2. A configuration file with no settings for memory and cpu - in this case, the memory and cpu resources will be calculated based on the underlying hardware instead of them being set to 8192 and 8 respectively. Isn't calculating the values from the hardware a better option? If people feel strongly about sticking to 8192 and 8, I don't have any problems changing them but it seems a bit odd.

          Show
          Varun Vasudev added a comment - Junping Du The old way to configure resource of NM is still useful, especially when there are other agents running (like: HBase RegionServer). Thus, user need flexibility to calculate resource themselves in some cases, so we should provide another new option instead of removing old way completely. The patch supports the old way. If a user has set values for memory and vcores, they're used without looking at the underlying hardware. I've added test cases to verify that behaviour as well. Have I missed a use case? Given this is a new feature, we shouldn't change cluster's behavior with old configuration in upgrade prospective. We should keep previous configuration work as usual especially when user use some default settings. There are two scenarios here - 1. A configuration file with custom settings for memory and cpu - nothing will change for these users. 2. A configuration file with no settings for memory and cpu - in this case, the memory and cpu resources will be calculated based on the underlying hardware instead of them being set to 8192 and 8 respectively. Isn't calculating the values from the hardware a better option? If people feel strongly about sticking to 8192 and 8, I don't have any problems changing them but it seems a bit odd.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12662475/apache-yarn-160.0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4664//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4664//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12662475/apache-yarn-160.0.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4664//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4664//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          The patch supports the old way.

          Thanks for clarification here. Yes. I saw the details of getYARNContainerMemoryMB() which sounds to honor previous NM resource configuration.

          Isn't calculating the values from the hardware a better option?

          Agree. But if the calculating results is not reasonable (like 0 or minus value), shall we use previous NM default value instead? At least, experienced users (especially with test purpose) already had some expectations even when they don't set any resource value here.

          Show
          Junping Du added a comment - The patch supports the old way. Thanks for clarification here. Yes. I saw the details of getYARNContainerMemoryMB() which sounds to honor previous NM resource configuration. Isn't calculating the values from the hardware a better option? Agree. But if the calculating results is not reasonable (like 0 or minus value), shall we use previous NM default value instead? At least, experienced users (especially with test purpose) already had some expectations even when they don't set any resource value here.
          Hide
          Varun Vasudev added a comment -

          Uploaded new patch to address Junping Du comments.

          Show
          Varun Vasudev added a comment - Uploaded new patch to address Junping Du comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12663362/apache-yarn-160.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          -1 release audit. The applied patch generated 3 release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4683//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-YARN-Build/4683//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4683//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12663362/apache-yarn-160.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 release audit . The applied patch generated 3 release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4683//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-YARN-Build/4683//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4683//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks for addressing my comments, Varun Vasudev!

          +  private static final Pattern PHYSICAL_ID_FORMAT =
          +      Pattern.compile("^physical id[ \t]*:[ \t]*([0-9]*)");
          +  private static final Pattern CORE_ID_FORMAT =
          +      Pattern.compile("^core id[ \t]*:[ \t]*([0-9]*)");
          

          Both physical id and core id are not guaranteed to have in /proc/cpuinfo (please see below for my local VM's info). We may use processor number instead in case these ids are 0 (like we did in Windows). Again, this weak my confidence that this automatic way of getting CPU/memory resources should happen by default (not sure if any cross-platform issues). May be a safer way here is to keep previous default behavior (with some static setting) with an extra config to enable this. We can wait this feature to be more stable later to change the default behavior.

          processor	: 0
          vendor_id	: GenuineIntel
          cpu family	: 6
          model		: 70
          model name	: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
          stepping	: 1
          cpu MHz		: 2295.265
          cache size	: 6144 KB
          fpu		: yes
          fpu_exception	: yes
          cpuid level	: 13
          wp		: yes
          flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi ept vpid fsgsbase smep
          bogomips	: 4590.53
          clflush size	: 64
          cache_alignment	: 64
          address sizes	: 40 bits physical, 48 bits virtual
          power management:
          
          Show
          Junping Du added a comment - Thanks for addressing my comments, Varun Vasudev ! + private static final Pattern PHYSICAL_ID_FORMAT = + Pattern.compile( "^physical id[ \t]*:[ \t]*([0-9]*)" ); + private static final Pattern CORE_ID_FORMAT = + Pattern.compile( "^core id[ \t]*:[ \t]*([0-9]*)" ); Both physical id and core id are not guaranteed to have in /proc/cpuinfo (please see below for my local VM's info). We may use processor number instead in case these ids are 0 (like we did in Windows). Again, this weak my confidence that this automatic way of getting CPU/memory resources should happen by default (not sure if any cross-platform issues). May be a safer way here is to keep previous default behavior (with some static setting) with an extra config to enable this. We can wait this feature to be more stable later to change the default behavior. processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 70 model name : Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz stepping : 1 cpu MHz : 2295.265 cache size : 6144 KB fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi ept vpid fsgsbase smep bogomips : 4590.53 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management:
          Hide
          Varun Vasudev added a comment -

          Junping Du

          Both physical id and core id are not guaranteed to have in /proc/cpuinfo (please see below for my local VM's info). We may use processor number instead in case these ids are 0 (like we did in Windows). Again, this weak my confidence that this automatic way of getting CPU/memory resources should happen by default (not sure if any cross-platform issues). May be a safer way here is to keep previous default behavior (with some static setting) with an extra config to enable this. We can wait this feature to be more stable later to change the default behavior.

          processor	: 0
          vendor_id	: GenuineIntel
          cpu family	: 6
          model		: 70
          model name	: Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
          stepping	: 1
          cpu MHz		: 2295.265
          cache size	: 6144 KB
          fpu		: yes
          fpu_exception	: yes
          cpuid level	: 13
          wp		: yes
          flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi ept vpid fsgsbase smep
          bogomips	: 4590.53
          clflush size	: 64
          cache_alignment	: 64
          address sizes	: 40 bits physical, 48 bits virtual
          power management:
          

          In the example you gave, where we have processors listed but no physical id or core id entries, the numProcessors will be set to the number of entries and numCores will be set to 1. From the diff -

          +      numCores = 1;
          

          There is also a test case to ensure this behaviour.

          In addition, cluster administrators can decide whether the NodeManager should report numProcessors or numCores by toggling yarn.nodemanager.resource.count-logical-processors-as-vcores which by default is true. In the vm example, by default the NodeManager will report vcores as the number of processor entries in /proc/cpuinfo. If yarn.nodemanager.resource.count-logical-processors-as-vcores is set to false, the NodeManager will report vcores as 1(if there are no physical id or core id entries).

          Show
          Varun Vasudev added a comment - Junping Du Both physical id and core id are not guaranteed to have in /proc/cpuinfo (please see below for my local VM's info). We may use processor number instead in case these ids are 0 (like we did in Windows). Again, this weak my confidence that this automatic way of getting CPU/memory resources should happen by default (not sure if any cross-platform issues). May be a safer way here is to keep previous default behavior (with some static setting) with an extra config to enable this. We can wait this feature to be more stable later to change the default behavior. processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 70 model name : Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz stepping : 1 cpu MHz : 2295.265 cache size : 6144 KB fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc up arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm ida arat epb xsaveopt pln pts dts tpr_shadow vnmi ept vpid fsgsbase smep bogomips : 4590.53 clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: In the example you gave, where we have processors listed but no physical id or core id entries, the numProcessors will be set to the number of entries and numCores will be set to 1. From the diff - + numCores = 1; There is also a test case to ensure this behaviour. In addition, cluster administrators can decide whether the NodeManager should report numProcessors or numCores by toggling yarn.nodemanager.resource.count-logical-processors-as-vcores which by default is true. In the vm example, by default the NodeManager will report vcores as the number of processor entries in /proc/cpuinfo. If yarn.nodemanager.resource.count-logical-processors-as-vcores is set to false, the NodeManager will report vcores as 1(if there are no physical id or core id entries).
          Hide
          Varun Vasudev added a comment -

          Comments from Jason Lowe in YARN-2440 about this feature led to some more changes. The latest patch introduces some new config variables
          1. yarn.nodemanager.containers-cpu-cores - the number of cores to be used for yarn containers. By default we use all cores.
          2. yarn.nodemanager.containers-cpu-percentage - the percentage of overall cpu to be used for yarn containers. By default we use all CPU.
          3. yarn.nodemanager.pcores-vcores-multiplier - a multiplier to convert pcores to vcores. By default it is 1. This can be used on clusters with heterogeneous hardware to have more containers run on faster CPUs.
          4. yarn.nodemanager.count-logical-processors-as-cores - flag to determine if hperthreads should be counted as cores. By default it is true.

          There's a some code between YARN-2440 and this patch. Depending on which one gets committed first, I'll change the patch appropriately.

          Show
          Varun Vasudev added a comment - Comments from Jason Lowe in YARN-2440 about this feature led to some more changes. The latest patch introduces some new config variables 1. yarn.nodemanager.containers-cpu-cores - the number of cores to be used for yarn containers. By default we use all cores. 2. yarn.nodemanager.containers-cpu-percentage - the percentage of overall cpu to be used for yarn containers. By default we use all CPU. 3. yarn.nodemanager.pcores-vcores-multiplier - a multiplier to convert pcores to vcores. By default it is 1. This can be used on clusters with heterogeneous hardware to have more containers run on faster CPUs. 4. yarn.nodemanager.count-logical-processors-as-cores - flag to determine if hperthreads should be counted as cores. By default it is true. There's a some code between YARN-2440 and this patch. Depending on which one gets committed first, I'll change the patch appropriately.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664137/apache-yarn-160.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4721//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4721//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664137/apache-yarn-160.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4721//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4721//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12664137/apache-yarn-160.2.patch
          against trunk revision 1556f86.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5953//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12664137/apache-yarn-160.2.patch against trunk revision 1556f86. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5953//console This message is automatically generated.
          Hide
          Varun Vasudev added a comment -

          Uploaded a new patch - apache-yarn-160.3.patch.
          1. rebase to trunk
          2. add a flag that allows users to turn off detection of underlying hardware.

          Show
          Varun Vasudev added a comment - Uploaded a new patch - apache-yarn-160.3.patch. 1. rebase to trunk 2. add a flag that allows users to turn off detection of underlying hardware.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12690562/apache-yarn-160.3.patch
          against trunk revision 788ee35.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 29 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6269//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6269//artifact/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6269//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12690562/apache-yarn-160.3.patch against trunk revision 788ee35. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 29 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-tools/hadoop-gridmix hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6269//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/6269//artifact/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6269//console This message is automatically generated.
          Hide
          Varun Vasudev added a comment -

          The findbugs warnings are unrelated to the patch.

          Show
          Varun Vasudev added a comment - The findbugs warnings are unrelated to the patch.
          Hide
          Allen Wittenauer added a comment -

          RAM-2*HADOOP_HEAPSIZE

          HADOOP_HEAPSIZE_MAX in trunk. HADOOP_HEAPSIZE was deprecated.

          Show
          Allen Wittenauer added a comment - RAM-2*HADOOP_HEAPSIZE HADOOP_HEAPSIZE_MAX in trunk. HADOOP_HEAPSIZE was deprecated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Quick comments on the patch:

          • LinuxResourceCalculatorPlugin: numPhysicalSockets is not used anywhere?
          • WindowsResourceCalculatorPlugin: Why is num-cores set = num-processors ?
          • yarn-default.xml: Change "it will set the X to Y" to be "it will set X to Y by default"
          • yarn.nodemanager.count-logical-processors-as-cores: Not sure of the use for this. On Linux, shouldn't we simply use the the returned numCores if they are valid? And fall-back to numProcessors?
          • yarn.nodemanager.enable-hardware-capability-detection: I think specifying the capabilities to be -1 is already a way to trigger this automatic detection, let's simply drop the flag and assume it to be true all the time?
          • CGroupsLCEResourceHandler: The log message 'LOG.info("node vcores = " + nodeVCores);' is printed for every container launch.
          • Should we enforce somewhere that numCores >= numProcessors if not that it is always a multiple?
                 int containerPhysicalMemoryMB =
                      (int) (0.8f * (physicalMemoryMB - (2 * hadoopHeapSizeMB)));
          

          We already have resource.percentage-physical-cpu-limit for CPUs - YARN-2440. How about simply adding a resource.percentage-pmem-limit instead making it a magic number in the code? Of course, we can have a default reserved percentage.

          Show
          Vinod Kumar Vavilapalli added a comment - Quick comments on the patch: LinuxResourceCalculatorPlugin: numPhysicalSockets is not used anywhere? WindowsResourceCalculatorPlugin: Why is num-cores set = num-processors ? yarn-default.xml: Change "it will set the X to Y" to be "it will set X to Y by default" yarn.nodemanager.count-logical-processors-as-cores: Not sure of the use for this. On Linux, shouldn't we simply use the the returned numCores if they are valid? And fall-back to numProcessors? yarn.nodemanager.enable-hardware-capability-detection: I think specifying the capabilities to be -1 is already a way to trigger this automatic detection, let's simply drop the flag and assume it to be true all the time? CGroupsLCEResourceHandler: The log message 'LOG.info("node vcores = " + nodeVCores);' is printed for every container launch. Should we enforce somewhere that numCores >= numProcessors if not that it is always a multiple? int containerPhysicalMemoryMB = ( int ) (0.8f * (physicalMemoryMB - (2 * hadoopHeapSizeMB))); We already have resource.percentage-physical-cpu-limit for CPUs - YARN-2440 . How about simply adding a resource.percentage-pmem-limit instead making it a magic number in the code? Of course, we can have a default reserved percentage.
          Hide
          Varun Vasudev added a comment -

          HADOOP_HEAPSIZE_MAX in trunk. HADOOP_HEAPSIZE was deprecated.

          Thanks for pointing this out Allen. I'll provide a patch for trunk and one for branch-2 when I address Vinod's comments.

          yarn.nodemanager.count-logical-processors-as-cores: Not sure of the use for this. On Linux, shouldn't we simply use the the returned numCores if they are valid? And fall-back to numProcessors?

          Some people prefer to count hyperthreads as a CPU and some don't. This lets users choose.

          yarn.nodemanager.enable-hardware-capability-detection: I think specifying the capabilities to be -1 is already a way to trigger this automatic detection, let's simply drop the flag and assume it to be true all the time?

          Junping felt we should add it to cover upgrade scenarios. What do you think?

          We already have resource.percentage-physical-cpu-limit for CPUs - YARN-2440. How about simply adding a resource.percentage-pmem-limit instead making it a magic number in the code? Of course, we can have a default reserved percentage.

          I think resource.percentage-pmem-limit should be analogous to resource.percentage-physical-cpu-limit in that it sets the limit as a percentage of total memory. What about something like "yarn.nodemanager.default-percentage-pmem-limit"?

          Show
          Varun Vasudev added a comment - HADOOP_HEAPSIZE_MAX in trunk. HADOOP_HEAPSIZE was deprecated. Thanks for pointing this out Allen. I'll provide a patch for trunk and one for branch-2 when I address Vinod's comments. yarn.nodemanager.count-logical-processors-as-cores: Not sure of the use for this. On Linux, shouldn't we simply use the the returned numCores if they are valid? And fall-back to numProcessors? Some people prefer to count hyperthreads as a CPU and some don't. This lets users choose. yarn.nodemanager.enable-hardware-capability-detection: I think specifying the capabilities to be -1 is already a way to trigger this automatic detection, let's simply drop the flag and assume it to be true all the time? Junping felt we should add it to cover upgrade scenarios. What do you think? We already have resource.percentage-physical-cpu-limit for CPUs - YARN-2440 . How about simply adding a resource.percentage-pmem-limit instead making it a magic number in the code? Of course, we can have a default reserved percentage. I think resource.percentage-pmem-limit should be analogous to resource.percentage-physical-cpu-limit in that it sets the limit as a percentage of total memory. What about something like "yarn.nodemanager.default-percentage-pmem-limit"?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Junping felt we should add it to cover upgrade scenarios. What do you think?

          Okay, not sure of the upgrade issue, but I think it makes sense to have an explicit flag.

          I think resource.percentage-pmem-limit should be analogous to resource.percentage-physical-cpu-limit in that it sets the limit as a percentage of total memory. What about something like "yarn.nodemanager.default-percentage-pmem-limit"?

          Don't quite understand your comment. We can have a resource.percentage-pmem-limit and say that this only works when hardware-detection is enabled?

          Show
          Vinod Kumar Vavilapalli added a comment - Junping felt we should add it to cover upgrade scenarios. What do you think? Okay, not sure of the upgrade issue, but I think it makes sense to have an explicit flag. I think resource.percentage-pmem-limit should be analogous to resource.percentage-physical-cpu-limit in that it sets the limit as a percentage of total memory. What about something like "yarn.nodemanager.default-percentage-pmem-limit"? Don't quite understand your comment. We can have a resource.percentage-pmem-limit and say that this only works when hardware-detection is enabled?
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12690562/apache-yarn-160.3.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / d3d019c
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7579/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12690562/apache-yarn-160.3.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / d3d019c Console output https://builds.apache.org/job/PreCommit-YARN-Build/7579/console This message was automatically generated.
          Hide
          Li Lu added a comment -

          The latest patch does not apply on trunk. Varun Vasudev would you mind to update it? Thanks! For now I'm canceling this patch.

          Show
          Li Lu added a comment - The latest patch does not apply on trunk. Varun Vasudev would you mind to update it? Thanks! For now I'm canceling this patch.
          Hide
          Varun Vasudev added a comment -

          Uploaded 005.patch which should apply to trunk and has auto detection of capability turned off by default. Also, change the config names to have the yarn.nodemanager.resource prefix.

          Show
          Varun Vasudev added a comment - Uploaded 005.patch which should apply to trunk and has auto detection of capability turned off by default. Also, change the config names to have the yarn.nodemanager.resource prefix.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 42s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 43s There were no new checkstyle issues.
          -1 whitespace 0m 27s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 tools/hadoop tests 14m 37s Tests passed in hadoop-gridmix.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 2s Tests passed in hadoop-yarn-server-nodemanager.
              65m 9s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732266/YARN-160.005.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / f4e2b3c
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/whitespace.txt
          hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-gridmix.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7890/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7890/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 42s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 43s There were no new checkstyle issues. -1 whitespace 0m 27s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 14m 37s Tests passed in hadoop-gridmix. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 2s Tests passed in hadoop-yarn-server-nodemanager.     65m 9s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732266/YARN-160.005.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / f4e2b3c whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/whitespace.txt hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-gridmix.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7890/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7890/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7890/console This message was automatically generated.
          Hide
          Varun Vasudev added a comment -

          Uploaded 006.patch to fix whitespace issues.

          Show
          Varun Vasudev added a comment - Uploaded 006.patch to fix whitespace issues.
          Hide
          Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 42s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 44s There were no new checkstyle issues.
          +1 whitespace 0m 27s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 31s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 tools/hadoop tests 14m 47s Tests passed in hadoop-gridmix.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.
              65m 21s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732288/YARN-160.006.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / 5c2f05c
          hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-gridmix.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7893/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7893/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 42s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 44s There were no new checkstyle issues. +1 whitespace 0m 27s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 31s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 14m 47s Tests passed in hadoop-gridmix. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 57s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 3s Tests passed in hadoop-yarn-server-nodemanager.     65m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732288/YARN-160.006.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 5c2f05c hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-gridmix.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7893/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7893/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7893/console This message was automatically generated.
          Hide
          Varun Vasudev added a comment -

          Uploaded 007.patch which improves some logging.

          Show
          Varun Vasudev added a comment - Uploaded 007.patch which improves some logging.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 57s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 7m 36s There were no new javac warning messages.
          +1 javadoc 10m 0s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 40s There were no new checkstyle issues.
          +1 whitespace 0m 29s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 33s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 tools/hadoop tests 15m 2s Tests passed in hadoop-gridmix.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 57s Tests failed in hadoop-yarn-common.
          -1 yarn tests 0m 17s Tests failed in hadoop-yarn-server-nodemanager.
              60m 30s  



          Reason Tests
          Failed unit tests hadoop.yarn.nodelabels.TestFileSystemNodeLabelsStore
          Failed build hadoop-yarn-server-nodemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732537/YARN-160.007.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / 92c38e4
          hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-gridmix.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7916/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7916/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 57s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 40s There were no new checkstyle issues. +1 whitespace 0m 29s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 33s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 tools/hadoop tests 15m 2s Tests passed in hadoop-gridmix. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 57s Tests failed in hadoop-yarn-common. -1 yarn tests 0m 17s Tests failed in hadoop-yarn-server-nodemanager.     60m 30s   Reason Tests Failed unit tests hadoop.yarn.nodelabels.TestFileSystemNodeLabelsStore Failed build hadoop-yarn-server-nodemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732537/YARN-160.007.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 92c38e4 hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-gridmix.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7916/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7916/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7916/console This message was automatically generated.
          Hide
          Varun Vasudev added a comment -

          The test failure is unrelated to the patch.

          Show
          Varun Vasudev added a comment - The test failure is unrelated to the patch.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The Windows related changes in ContainerExecutor in the last patch (were not there before) seem spurious. Earlier it was (containerVCores * 10000) / (configured-node-vcores * configured-yarn-cpu-percent). Now it is (containerVCores * 100 * configured-yarn-cpu-percent) / configured-node-vcores.

          Can you describe these changes?

          Also the patch has gone stale unfortunately. Canceling.

          Show
          Vinod Kumar Vavilapalli added a comment - The Windows related changes in ContainerExecutor in the last patch (were not there before) seem spurious. Earlier it was (containerVCores * 10000) / (configured-node-vcores * configured-yarn-cpu-percent). Now it is (containerVCores * 100 * configured-yarn-cpu-percent) / configured-node-vcores. Can you describe these changes? Also the patch has gone stale unfortunately. Canceling.
          Hide
          Varun Vasudev added a comment -

          Uploaded 008.patch, which should apply to trunk.

          Show
          Varun Vasudev added a comment - Uploaded 008.patch, which should apply to trunk.
          Hide
          Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 6s The applied patch generated 1 new checkstyle issues (total was 214, now 215).
          +1 whitespace 0m 28s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 tools/hadoop tests 14m 39s Tests passed in hadoop-gridmix.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common.
          +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.
              65m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735336/YARN-160.008.patch
          Optional Tests javac unit findbugs checkstyle javadoc
          git revision trunk / 022f49d
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-gridmix.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8085/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8085/console

          This message was automatically generated.

          Show
          Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 6s The applied patch generated 1 new checkstyle issues (total was 214, now 215). +1 whitespace 0m 28s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 4m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 tools/hadoop tests 14m 39s Tests passed in hadoop-gridmix. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 58s Tests passed in hadoop-yarn-common. +1 yarn tests 6m 8s Tests passed in hadoop-yarn-server-nodemanager.     65m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735336/YARN-160.008.patch Optional Tests javac unit findbugs checkstyle javadoc git revision trunk / 022f49d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-gridmix test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-gridmix.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8085/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8085/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8085/console This message was automatically generated.
          Hide
          Varun Vasudev added a comment -

          The change for the Windows cpu limits fixes a bug in the current implementation. The current implementation allows YARN containers to exceed the configured cpu limit in some cases.

          Show
          Varun Vasudev added a comment - The change for the Windows cpu limits fixes a bug in the current implementation. The current implementation allows YARN containers to exceed the configured cpu limit in some cases.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Tx for the explanation, Varun. The new logic definitely makes sense to me.

          The patch looks good. Checking this in.

          Show
          Vinod Kumar Vavilapalli added a comment - Tx for the explanation, Varun. The new logic definitely makes sense to me. The patch looks good. Checking this in.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk and branch-2. Thanks Varun Vasudev!

          We need a comprehensive doc after this patch explaining how to configure CPU/memory allocations on the NodeManager. Filed YARN-3720, please feel free to take it up.

          Show
          Vinod Kumar Vavilapalli added a comment - Committed this to trunk and branch-2. Thanks Varun Vasudev ! We need a comprehensive doc after this patch explaining how to configure CPU/memory allocations on the NodeManager. Filed YARN-3720 , please feel free to take it up.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7903/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7903 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7903/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/209/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/209/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #940 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/940/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #940 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/940/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/208/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #208 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/208/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2138 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2138/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2138 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2138/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #198 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/198/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #198 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/198/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2156 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2156/)
          YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2156 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2156/ ) YARN-160 . Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. (vinodkv: rev 500a1d9c76ec612b4e737888f4be79951c11591d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestLinuxResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/CgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestCgroupsLCEResourcesHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/WindowsResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestNodeManagerHardwareUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/NodeManagerHardwareUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java

            People

            • Assignee:
              Varun Vasudev
              Reporter:
              Alejandro Abdelnur
            • Votes:
              1 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development