Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5356

NodeManager should communicate physical resource capability to ResourceManager

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Currently ResourceUtilization contains absolute quantities of resource used (e.g. 4096MB memory used). It would be good if the NM also communicated the actual physical resource capabilities of the node so that the RM can use this data to schedule more effectively (overcommit, etc)

      Currently the only available information is the Resource the node registered with (or later updated using updateNodeResource). However, these aren't really sufficient to get a good view of how utilized a resource is. For example, if a node reports 400% CPU utilization, does that mean it's completely full, or barely utilized? Today there is no reliable way to figure this out.

      Íñigo Goiri - Lots of good work is happening in YARN-2965 so curious if you have thoughts/opinions on this?

      1. YARN-5356.011.patch
        26 kB
        Íñigo Goiri
      2. YARN-5356.010.patch
        23 kB
        Íñigo Goiri
      3. YARN-5356.009.patch
        23 kB
        Íñigo Goiri
      4. YARN-5356.008.patch
        23 kB
        Íñigo Goiri
      5. YARN-5356.007.patch
        20 kB
        Íñigo Goiri
      6. YARN-5356.006.patch
        20 kB
        Íñigo Goiri
      7. YARN-5356.005.patch
        20 kB
        Íñigo Goiri
      8. YARN-5356.004.patch
        20 kB
        Íñigo Goiri
      9. YARN-5356.003.patch
        19 kB
        Íñigo Goiri
      10. YARN-5356.002.patch
        17 kB
        Íñigo Goiri
      11. YARN-5356.002.patch
        17 kB
        Íñigo Goiri
      12. YARN-5356.001.patch
        13 kB
        Íñigo Goiri
      13. YARN-5356.000.patch
        9 kB
        Íñigo Goiri

        Issue Links

          Activity

          Hide
          elgoiri Íñigo Goiri added a comment - - edited

          In general, we have 3 values:

          • Actual resources of the full machine. This currently comes from NodeManagerHardwareUtils if I remember correctly. For example, it can be 12 cores.
          • Resource available for the Node Manager. This is currently defined in yarn-site.xml with key yarn.nodemanager.resource.cpu-vcores or with the updateNodeResource(). For example, 6 cores.
          • Actual utilization of the machine. This is extracted in the NodeResourceMonitor with the ResourceCalculatorPlugin. And it can be 400%, which would imply 4 out of the 12 cores used.

          Nathan Roberts, I understand that your problem is that with the current approach you know that you have 6 cores available to the NM and 4 of them are used. However, the machine is not that utilized (~30%). Correct? In that case, we would only need to report the actual size of the machine at registration time as it would never change. Not sure that ResourceUtilization would be the right place for that as it would be reported in every heartbeat continuously.

          Show
          elgoiri Íñigo Goiri added a comment - - edited In general, we have 3 values: Actual resources of the full machine. This currently comes from NodeManagerHardwareUtils if I remember correctly. For example, it can be 12 cores. Resource available for the Node Manager. This is currently defined in yarn-site.xml with key yarn.nodemanager.resource.cpu-vcores or with the updateNodeResource() . For example, 6 cores. Actual utilization of the machine. This is extracted in the NodeResourceMonitor with the ResourceCalculatorPlugin . And it can be 400%, which would imply 4 out of the 12 cores used. Nathan Roberts , I understand that your problem is that with the current approach you know that you have 6 cores available to the NM and 4 of them are used. However, the machine is not that utilized (~30%). Correct? In that case, we would only need to report the actual size of the machine at registration time as it would never change. Not sure that ResourceUtilization would be the right place for that as it would be reported in every heartbeat continuously.
          Hide
          nroberts Nathan Roberts added a comment - - edited

          Nathan Roberts, I understand that your problem is that with the current approach you know that you have 6 cores available to the NM and 4 of them are used. However, the machine is not that utilized (~30%). Correct? In that case, we would only need to report the actual size of the machine at registration time as it would never change. Not sure that ResourceUtilization would be the right place for that as it would be reported in every heartbeat continuously.

          Íñigo Goiri, Yep, that's exactly correct. I think reporting the physical capabilities of the machine during registration should be ok. At least with linux it is technically possible for the machine to change (e.g. echo 0 > /sys/devices/system/cpu/cpu3/online, OR memory gets automatically removed because it's getting ECC errors, OR something reserves a bunch of memory for huge pages, OR NIC re-negotiates from 10G to 1G), but I think these might be unusual enough that we could ignore them. I originally suggested tweaking ResourceUtilization due to this small chance of a physical resource changing but am happy to go either way.

          Show
          nroberts Nathan Roberts added a comment - - edited Nathan Roberts, I understand that your problem is that with the current approach you know that you have 6 cores available to the NM and 4 of them are used. However, the machine is not that utilized (~30%). Correct? In that case, we would only need to report the actual size of the machine at registration time as it would never change. Not sure that ResourceUtilization would be the right place for that as it would be reported in every heartbeat continuously. Íñigo Goiri , Yep, that's exactly correct. I think reporting the physical capabilities of the machine during registration should be ok. At least with linux it is technically possible for the machine to change (e.g. echo 0 > /sys/devices/system/cpu/cpu3/online, OR memory gets automatically removed because it's getting ECC errors, OR something reserves a bunch of memory for huge pages, OR NIC re-negotiates from 10G to 1G), but I think these might be unusual enough that we could ignore them. I originally suggested tweaking ResourceUtilization due to this small chance of a physical resource changing but am happy to go either way.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Nathan Roberts, I think that changing machine resources it's not that common and admins could always restart the Node Manager. in that case, I would just extend RegisterNodeManagerRequestProto in yarn_server_common_service_protos.proto and populate it in NodeStatusUpdaterImpl#registerWithRM(). If we don't care about reporting network, then adding Resource is fine. However, if we go into network and disk, ResourceUtilization has those fields but I don't think that the semantics we want to provide matches with resource utilization.

          I can post a patch with these changes if you want.

          Show
          elgoiri Íñigo Goiri added a comment - Nathan Roberts , I think that changing machine resources it's not that common and admins could always restart the Node Manager. in that case, I would just extend RegisterNodeManagerRequestProto in yarn_server_common_service_protos.proto and populate it in NodeStatusUpdaterImpl#registerWithRM() . If we don't care about reporting network, then adding Resource is fine. However, if we go into network and disk, ResourceUtilization has those fields but I don't think that the semantics we want to provide matches with resource utilization. I can post a patch with these changes if you want.
          Hide
          nroberts Nathan Roberts added a comment -

          I can post a patch with these changes if you want.

          That would be great. If not I can work on it later this week.

          Show
          nroberts Nathan Roberts added a comment - I can post a patch with these changes if you want. That would be great. If not I can work on it later this week.
          Hide
          elgoiri Íñigo Goiri added a comment -

          It looks like there is some work we may want to leverage in YARN-4081 where they have YARN-4081 to add multiple resources.
          I'd like to have some feedback from Varun Vasudev and Arun Suresh about their opinion on this.

          Show
          elgoiri Íñigo Goiri added a comment - It looks like there is some work we may want to leverage in YARN-4081 where they have YARN-4081 to add multiple resources. I'd like to have some feedback from Varun Vasudev and Arun Suresh about their opinion on this.
          Hide
          elgoiri Íñigo Goiri added a comment -

          First proposal for sending physical resources in the node to the RM.

          Show
          elgoiri Íñigo Goiri added a comment - First proposal for sending physical resources in the node to the RM.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 32s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 6m 40s trunk passed
          +1 compile 1m 30s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 0m 32s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          -1 mvninstall 0m 16s hadoop-yarn-server-common in the patch failed.
          -1 mvninstall 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
          -1 compile 0m 17s hadoop-yarn-server in the patch failed.
          -1 cc 0m 17s hadoop-yarn-server in the patch failed.
          -1 javac 0m 17s hadoop-yarn-server in the patch failed.
          -1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 43 unchanged - 0 fixed = 44 total (was 43)
          -1 mvnsite 0m 17s hadoop-yarn-server-common in the patch failed.
          -1 mvnsite 0m 16s hadoop-yarn-server-nodemanager in the patch failed.
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 11s hadoop-yarn-server-common in the patch failed.
          -1 findbugs 0m 11s hadoop-yarn-server-nodemanager in the patch failed.
          +1 javadoc 0m 27s the patch passed
          -1 unit 0m 16s hadoop-yarn-server-common in the patch failed.
          -1 unit 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          16m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817559/YARN-5356.000.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 027fa57e99d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5537c6b
          Default Java 1.8.0_91
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12333/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12333/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 1m 30s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 0m 32s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch -1 mvninstall 0m 16s hadoop-yarn-server-common in the patch failed. -1 mvninstall 0m 15s hadoop-yarn-server-nodemanager in the patch failed. -1 compile 0m 17s hadoop-yarn-server in the patch failed. -1 cc 0m 17s hadoop-yarn-server in the patch failed. -1 javac 0m 17s hadoop-yarn-server in the patch failed. -1 checkstyle 0m 27s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 43 unchanged - 0 fixed = 44 total (was 43) -1 mvnsite 0m 17s hadoop-yarn-server-common in the patch failed. -1 mvnsite 0m 16s hadoop-yarn-server-nodemanager in the patch failed. +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 11s hadoop-yarn-server-common in the patch failed. -1 findbugs 0m 11s hadoop-yarn-server-nodemanager in the patch failed. +1 javadoc 0m 27s the patch passed -1 unit 0m 16s hadoop-yarn-server-common in the patch failed. -1 unit 0m 15s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 16m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12817559/YARN-5356.000.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 027fa57e99d2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5537c6b Default Java 1.8.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12333/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12333/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/12333/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Íñigo Goiri for the patch! Some quick comments

          +    int nodeMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024);
          +    int nodeVirtualCores = rcp.getNumProcessors();
          +    this.nodeResource = Resource.newInstance(nodeMemoryMb, nodeVirtualCores);
          
          • Wondering if we should remove "Virtual" from variable names since these are real cores and the ratio of VCores to Cores isn't always 1. Another option might be "nodePhysicalCores"?
          +
          +  /**
          +   * Get the physical resources in the node to properly estimate resource
          +   * utilization.
          +   * @return Physical resources in the node.
          +   */
          +  public abstract Resource getNodeResource();
          +
          +  /**
          +   * Set the physical resources in the node to properly estimate resource
          +   * utilization.
          +   * @param nodeResource Physical resources in the node.
          +   */
          +  public abstract void setNodeResource(Resource nodeResource);
          
          • Difference between getResource() and getNodeResource() might lead to confusion. Maybe getPhysicalResource()?
          • Should we change RMNodeImpl so that it contains the physicalNodeResource?
          Show
          nroberts Nathan Roberts added a comment - Thanks Íñigo Goiri for the patch! Some quick comments + int nodeMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024); + int nodeVirtualCores = rcp.getNumProcessors(); + this.nodeResource = Resource.newInstance(nodeMemoryMb, nodeVirtualCores); Wondering if we should remove "Virtual" from variable names since these are real cores and the ratio of VCores to Cores isn't always 1. Another option might be "nodePhysicalCores"? + + /** + * Get the physical resources in the node to properly estimate resource + * utilization. + * @return Physical resources in the node. + */ + public abstract Resource getNodeResource(); + + /** + * Set the physical resources in the node to properly estimate resource + * utilization. + * @param nodeResource Physical resources in the node. + */ + public abstract void setNodeResource(Resource nodeResource); Difference between getResource() and getNodeResource() might lead to confusion. Maybe getPhysicalResource()? Should we change RMNodeImpl so that it contains the physicalNodeResource?
          Hide
          elgoiri Íñigo Goiri added a comment -

          Renaming to physical resources and adding to RMNode.

          Show
          elgoiri Íñigo Goiri added a comment - Renaming to physical resources and adding to RMNode.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 12m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 8s Maven dependency ordering for branch
          +1 mvninstall 7m 8s trunk passed
          +1 compile 1m 35s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 29s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 2m 19s trunk passed
          +1 javadoc 0m 53s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          -1 mvninstall 0m 15s hadoop-yarn-server-common in the patch failed.
          -1 mvninstall 0m 16s hadoop-yarn-server-nodemanager in the patch failed.
          -1 mvninstall 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 compile 0m 15s hadoop-yarn-server in the patch failed.
          -1 cc 0m 15s hadoop-yarn-server in the patch failed.
          -1 javac 0m 15s hadoop-yarn-server in the patch failed.
          -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 128 unchanged - 1 fixed = 132 total (was 129)
          -1 mvnsite 0m 15s hadoop-yarn-server-common in the patch failed.
          -1 mvnsite 0m 17s hadoop-yarn-server-nodemanager in the patch failed.
          -1 mvnsite 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 10s hadoop-yarn-server-common in the patch failed.
          -1 findbugs 0m 11s hadoop-yarn-server-nodemanager in the patch failed.
          -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 javadoc 0m 45s the patch passed
          -1 unit 0m 15s hadoop-yarn-server-common in the patch failed.
          -1 unit 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
          -1 unit 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          33m 25s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818596/YARN-5356.001.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux ddb1df2af966 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 76be82b
          Default Java 1.8.0_91
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12356/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12356/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 8s Maven dependency ordering for branch +1 mvninstall 7m 8s trunk passed +1 compile 1m 35s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 29s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 19s trunk passed +1 javadoc 0m 53s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch -1 mvninstall 0m 15s hadoop-yarn-server-common in the patch failed. -1 mvninstall 0m 16s hadoop-yarn-server-nodemanager in the patch failed. -1 mvninstall 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 0m 15s hadoop-yarn-server in the patch failed. -1 cc 0m 15s hadoop-yarn-server in the patch failed. -1 javac 0m 15s hadoop-yarn-server in the patch failed. -1 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 128 unchanged - 1 fixed = 132 total (was 129) -1 mvnsite 0m 15s hadoop-yarn-server-common in the patch failed. -1 mvnsite 0m 17s hadoop-yarn-server-nodemanager in the patch failed. -1 mvnsite 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 10s hadoop-yarn-server-common in the patch failed. -1 findbugs 0m 11s hadoop-yarn-server-nodemanager in the patch failed. -1 findbugs 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 45s the patch passed -1 unit 0m 15s hadoop-yarn-server-common in the patch failed. -1 unit 0m 15s hadoop-yarn-server-nodemanager in the patch failed. -1 unit 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 33m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818596/YARN-5356.001.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux ddb1df2af966 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 76be82b Default Java 1.8.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12356/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/12356/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixed compilation issue and checkstyle.

          Show
          elgoiri Íñigo Goiri added a comment - Fixed compilation issue and checkstyle.
          Hide
          nroberts Nathan Roberts added a comment -

          Hi Inigo, couple more comments when looking over most recent patch:

          +    int physicalMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024);
          
          • Think this needs to be (int)(rcp.getPhysicalMemorySize() / (1024 * 1024))
          • Would be good to get some other comments on the approach. Once we have general agreement then probably need to add a couple of tests.

          Thanks again.

          Show
          nroberts Nathan Roberts added a comment - Hi Inigo, couple more comments when looking over most recent patch: + int physicalMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024); Think this needs to be (int)(rcp.getPhysicalMemorySize() / (1024 * 1024)) Would be good to get some other comments on the approach. Once we have general agreement then probably need to add a couple of tests. Thanks again.
          Hide
          elgoiri Íñigo Goiri added a comment -

          I agree on getting more people to check this approach, Karthik Kambatla, Tan, Wangda?

          I thought about the unit test a little and I think we could collect some fake physical resources and then check that they make it into the RM.

          Show
          elgoiri Íñigo Goiri added a comment - I agree on getting more people to check this approach, Karthik Kambatla , Tan, Wangda ? I thought about the unit test a little and I think we could collect some fake physical resources and then check that they make it into the RM.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 4s trunk passed
          +1 compile 1m 33s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 1m 32s trunk passed
          +1 mvneclipse 0m 43s trunk passed
          +1 findbugs 2m 15s trunk passed
          +1 javadoc 0m 52s trunk passed
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 9s the patch passed
          +1 compile 1m 33s the patch passed
          +1 cc 1m 33s the patch passed
          +1 javac 1m 33s the patch passed
          -1 checkstyle 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 139 unchanged - 3 fixed = 143 total (was 142)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 35s the patch passed
          +1 javadoc 0m 46s the patch passed
          +1 unit 0m 25s hadoop-yarn-server-common in the patch passed.
          -1 unit 13m 10s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 37m 43s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          78m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818659/YARN-5356.002.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 7a29030628ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 92fe2db
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12363/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12363/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 17s Maven dependency ordering for branch +1 mvninstall 7m 4s trunk passed +1 compile 1m 33s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 2m 15s trunk passed +1 javadoc 0m 52s trunk passed 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 9s the patch passed +1 compile 1m 33s the patch passed +1 cc 1m 33s the patch passed +1 javac 1m 33s the patch passed -1 checkstyle 1m 28s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 139 unchanged - 3 fixed = 143 total (was 142) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 35s the patch passed +1 javadoc 0m 46s the patch passed +1 unit 0m 25s hadoop-yarn-server-common in the patch passed. -1 unit 13m 10s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 37m 43s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 78m 44s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.TestDirectoryCollection Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818659/YARN-5356.002.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7a29030628ad 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 92fe2db Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12363/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12363/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/12363/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Hi Íñigo Goiri. Tried out the patch but get NPE in RM because physicalResource is null. Think this code in org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RegisterNodeManagerRequestPBImpl.setPhysicalResource needs to set it via the builder as well.

            @Override
            public synchronized void setPhysicalResource(Resource pPhysicalResource) {
              maybeInitBuilder();
              if (pPhysicalResource == null) {
                builder.clearPhysicalResource();
              }
              this.physicalResource = pPhysicalResource;
            }
          
          Show
          nroberts Nathan Roberts added a comment - Hi Íñigo Goiri . Tried out the patch but get NPE in RM because physicalResource is null. Think this code in org.apache.hadoop.yarn.server.api.protocolrecords.impl.pb.RegisterNodeManagerRequestPBImpl.setPhysicalResource needs to set it via the builder as well. @Override public synchronized void setPhysicalResource(Resource pPhysicalResource) { maybeInitBuilder(); if (pPhysicalResource == null ) { builder.clearPhysicalResource(); } this .physicalResource = pPhysicalResource; }
          Hide
          elgoiri Íñigo Goiri added a comment -

          True, this is not done properly. I need to add a unit test for the PB too. I'll post a patch soon.

          Show
          elgoiri Íñigo Goiri added a comment - True, this is not done properly. I need to add a unit test for the PB too. I'll post a patch soon.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing NPE in PB for physical resources.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing NPE in PB for physical resources.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing NPE in PB.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing NPE in PB.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 57s Maven dependency ordering for branch
          +1 mvninstall 8m 23s trunk passed
          +1 compile 1m 46s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 38s trunk passed
          +1 mvneclipse 0m 46s trunk passed
          +1 findbugs 2m 38s trunk passed
          -1 javadoc 0m 24s hadoop-yarn-server-resourcemanager in trunk failed.
          0 mvndep 0m 9s Maven dependency ordering for patch
          -1 mvninstall 0m 32s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 compile 1m 19s hadoop-yarn-server in the patch failed.
          -1 cc 1m 19s hadoop-yarn-server in the patch failed.
          -1 javac 1m 19s hadoop-yarn-server in the patch failed.
          -1 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 151 unchanged - 3 fixed = 155 total (was 154)
          -1 mvnsite 0m 32s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 mvneclipse 0m 39s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 27s hadoop-yarn-server-resourcemanager in the patch failed.
          -1 javadoc 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 unit 15m 1s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 0m 37s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          43m 21s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829312/YARN-5356.003.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 96e202fff39e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 98bdb51
          Default Java 1.8.0_101
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13158/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13158/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 57s Maven dependency ordering for branch +1 mvninstall 8m 23s trunk passed +1 compile 1m 46s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 38s trunk passed +1 mvneclipse 0m 46s trunk passed +1 findbugs 2m 38s trunk passed -1 javadoc 0m 24s hadoop-yarn-server-resourcemanager in trunk failed. 0 mvndep 0m 9s Maven dependency ordering for patch -1 mvninstall 0m 32s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 1m 19s hadoop-yarn-server in the patch failed. -1 cc 1m 19s hadoop-yarn-server in the patch failed. -1 javac 1m 19s hadoop-yarn-server in the patch failed. -1 checkstyle 0m 31s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 4 new + 151 unchanged - 3 fixed = 155 total (was 154) -1 mvnsite 0m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 27s hadoop-yarn-server-resourcemanager in the patch failed. -1 javadoc 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 unit 15m 1s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 0m 37s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 43m 21s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829312/YARN-5356.003.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 96e202fff39e 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 98bdb51 Default Java 1.8.0_101 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13158/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13158/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server Console output https://builds.apache.org/job/PreCommit-YARN-Build/13158/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing compilation.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing compilation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 12s trunk passed
          +1 compile 7m 2s trunk passed
          +1 checkstyle 1m 30s trunk passed
          +1 mvnsite 1m 51s trunk passed
          +1 mvneclipse 0m 58s trunk passed
          +1 findbugs 2m 48s trunk passed
          +1 javadoc 1m 7s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 32s the patch passed
          +1 compile 7m 1s the patch passed
          +1 cc 7m 1s the patch passed
          +1 javac 7m 1s the patch passed
          -1 checkstyle 1m 29s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164)
          +1 mvnsite 1m 52s the patch passed
          +1 mvneclipse 0m 57s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 16s the patch passed
          +1 javadoc 1m 6s the patch passed
          +1 unit 0m 27s hadoop-yarn-server-common in the patch passed.
          +1 unit 14m 35s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 33m 52s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 1m 1s hadoop-sls in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          91m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829425/YARN-5356.004.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux f586f1d5482d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 9f03b40
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13165/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13165/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13165/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 12s trunk passed +1 compile 7m 2s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 2m 48s trunk passed +1 javadoc 1m 7s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 7m 1s the patch passed +1 cc 7m 1s the patch passed +1 javac 7m 1s the patch passed -1 checkstyle 1m 29s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 0m 27s hadoop-yarn-server-common in the patch passed. +1 unit 14m 35s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 33m 52s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 1m 1s hadoop-sls in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 91m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12829425/YARN-5356.004.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux f586f1d5482d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9f03b40 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13165/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13165/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13165/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          kshukla Kuhu Shukla added a comment -

          Running TestNodeStatusUpdaterForLabels#testInvalidNodeLabelsFromProvider,
          testNodeStatusUpdaterForNodeLabels
          on MAC causes NullPointerException as the ResourcePlugin returns null,

          We should protect against this in NodeStatusUpdaterImpl#serviceInit:

          // Get actual node physical resources
              ResourceCalculatorPlugin rcp =
                  ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf);
              int physicalMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024);
          
          Show
          kshukla Kuhu Shukla added a comment - Running TestNodeStatusUpdaterForLabels#testInvalidNodeLabelsFromProvider , testNodeStatusUpdaterForNodeLabels on MAC causes NullPointerException as the ResourcePlugin returns null, We should protect against this in NodeStatusUpdaterImpl#serviceInit: // Get actual node physical resources ResourceCalculatorPlugin rcp = ResourceCalculatorPlugin.getResourceCalculatorPlugin( null , conf); int physicalMemoryMb = ( int ) rcp.getPhysicalMemorySize() / (1024 * 1024);
          Hide
          elgoiri Íñigo Goiri added a comment -

          Makes sense. Not sure what to do in this case though:

          1. Leave physicalResource as null
          2. Set physicalResource to no resources
            I would go for the first one but I'm open to suggestions.
          Show
          elgoiri Íñigo Goiri added a comment - Makes sense. Not sure what to do in this case though: Leave physicalResource as null Set physicalResource to no resources I would go for the first one but I'm open to suggestions.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing NullPointerException when resource calculator not available.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing NullPointerException when resource calculator not available.
          Hide
          kshukla Kuhu Shukla added a comment -

          I think that if we do not have the values from the plugin we could initialize the physicalResource to totalNodeResource (from conf values) instead of zero. May be something like :

              int physicalMemoryMb;
              int physicalCores;
              if (rcp != null) {
                physicalMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024);
                physicalCores = rcp.getNumProcessors();
              } else {
                physicalMemoryMb = conf.getInt(
                    YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB)
                    + conf.getInt(
                    YarnConfiguration.NM_SYSTEM_RESERVED_PMEM_MB, 0);
                ......
              }
          
          Show
          kshukla Kuhu Shukla added a comment - I think that if we do not have the values from the plugin we could initialize the physicalResource to totalNodeResource (from conf values) instead of zero. May be something like : int physicalMemoryMb; int physicalCores; if (rcp != null ) { physicalMemoryMb = ( int ) rcp.getPhysicalMemorySize() / (1024 * 1024); physicalCores = rcp.getNumProcessors(); } else { physicalMemoryMb = conf.getInt( YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB) + conf.getInt( YarnConfiguration.NM_SYSTEM_RESERVED_PMEM_MB, 0); ...... }
          Hide
          kshukla Kuhu Shukla added a comment -

          I think that if we do not have the values from the plugin we could initialize the physicalResource to totalNodeResource (from conf values) instead of zero. May be something like :

              int physicalMemoryMb;
              int physicalCores;
              if (rcp != null) {
                physicalMemoryMb = (int) rcp.getPhysicalMemorySize() / (1024 * 1024);
                physicalCores = rcp.getNumProcessors();
              } else {
                physicalMemoryMb = conf.getInt(
                    YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB)
                    + conf.getInt(
                    YarnConfiguration.NM_SYSTEM_RESERVED_PMEM_MB, 0);
                ......
              }
          
          Show
          kshukla Kuhu Shukla added a comment - I think that if we do not have the values from the plugin we could initialize the physicalResource to totalNodeResource (from conf values) instead of zero. May be something like : int physicalMemoryMb; int physicalCores; if (rcp != null ) { physicalMemoryMb = ( int ) rcp.getPhysicalMemorySize() / (1024 * 1024); physicalCores = rcp.getNumProcessors(); } else { physicalMemoryMb = conf.getInt( YarnConfiguration.NM_PMEM_MB, YarnConfiguration.DEFAULT_NM_PMEM_MB) + conf.getInt( YarnConfiguration.NM_SYSTEM_RESERVED_PMEM_MB, 0); ...... }
          Hide
          elgoiri Íñigo Goiri added a comment -

          OK, that makes sense too. Let me switch to that.

          Show
          elgoiri Íñigo Goiri added a comment - OK, that makes sense too. Let me switch to that.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Using configured capacity when physical is not available.

          Show
          elgoiri Íñigo Goiri added a comment - Using configured capacity when physical is not available.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 45s Maven dependency ordering for branch
          +1 mvninstall 9m 28s trunk passed
          +1 compile 7m 56s trunk passed
          +1 checkstyle 1m 32s trunk passed
          +1 mvnsite 1m 54s trunk passed
          +1 mvneclipse 0m 59s trunk passed
          +1 findbugs 2m 43s trunk passed
          +1 javadoc 1m 6s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 6m 51s the patch passed
          +1 cc 6m 51s the patch passed
          +1 javac 6m 51s the patch passed
          -1 checkstyle 1m 29s root: The patch generated 4 new + 160 unchanged - 3 fixed = 164 total (was 163)
          +1 mvnsite 1m 50s the patch passed
          +1 mvneclipse 0m 59s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 15s the patch passed
          +1 javadoc 1m 6s the patch passed
          +1 unit 0m 27s hadoop-yarn-server-common in the patch passed.
          +1 unit 15m 1s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 39m 2s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 1m 37s hadoop-sls in the patch passed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          102m 50s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834471/YARN-5356.005.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux cabfe3866fce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6d2da38
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13453/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13453/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 9m 28s trunk passed +1 compile 7m 56s trunk passed +1 checkstyle 1m 32s trunk passed +1 mvnsite 1m 54s trunk passed +1 mvneclipse 0m 59s trunk passed +1 findbugs 2m 43s trunk passed +1 javadoc 1m 6s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 6m 51s the patch passed +1 cc 6m 51s the patch passed +1 javac 6m 51s the patch passed -1 checkstyle 1m 29s root: The patch generated 4 new + 160 unchanged - 3 fixed = 164 total (was 163) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 59s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 15s the patch passed +1 javadoc 1m 6s the patch passed +1 unit 0m 27s hadoop-yarn-server-common in the patch passed. +1 unit 15m 1s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 39m 2s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 37s hadoop-sls in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 102m 50s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834471/YARN-5356.005.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux cabfe3866fce 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6d2da38 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13453/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13453/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13453/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 40s Maven dependency ordering for branch
          +1 mvninstall 7m 3s trunk passed
          +1 compile 6m 50s trunk passed
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 1m 49s trunk passed
          +1 mvneclipse 0m 57s trunk passed
          +1 findbugs 2m 37s trunk passed
          +1 javadoc 1m 4s trunk passed
          0 mvndep 0m 14s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 6m 43s the patch passed
          +1 cc 6m 43s the patch passed
          +1 javac 6m 43s the patch passed
          -1 checkstyle 1m 28s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 57s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 9s the patch passed
          +1 javadoc 1m 4s the patch passed
          +1 unit 0m 26s hadoop-yarn-server-common in the patch passed.
          +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 35m 38s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 0m 54s hadoop-sls in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          93m 52s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834522/YARN-5356.006.patch
          JIRA Issue YARN-5356
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux fd527e52320b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6d2da38
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13454/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13454/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 40s Maven dependency ordering for branch +1 mvninstall 7m 3s trunk passed +1 compile 6m 50s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 1m 49s trunk passed +1 mvneclipse 0m 57s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 4s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 6m 43s the patch passed +1 cc 6m 43s the patch passed +1 javac 6m 43s the patch passed -1 checkstyle 1m 28s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 9s the patch passed +1 javadoc 1m 4s the patch passed +1 unit 0m 26s hadoop-yarn-server-common in the patch passed. +1 unit 14m 56s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 35m 38s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 0m 54s hadoop-sls in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 93m 52s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12834522/YARN-5356.006.patch JIRA Issue YARN-5356 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux fd527e52320b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6d2da38 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13454/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13454/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13454/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Hi Íñigo Goiri. Looked over version 6 of the patch. I am wondering about the comment regarding order-of-operations at: https://issues.apache.org/jira/browse/YARN-5356?focusedCommentId=15383184&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15383184

          I think it's incorrect because we need the cast to happen after we've scaled down to MB, but right now I think it's happening on the result of getPhysicalMemorySize(). LMK if I'm not in sync on this.

          Show
          nroberts Nathan Roberts added a comment - Hi Íñigo Goiri . Looked over version 6 of the patch. I am wondering about the comment regarding order-of-operations at: https://issues.apache.org/jira/browse/YARN-5356?focusedCommentId=15383184&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15383184 I think it's incorrect because we need the cast to happen after we've scaled down to MB, but right now I think it's happening on the result of getPhysicalMemorySize(). LMK if I'm not in sync on this.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing rounding issue.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing rounding issue.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Íñigo Goiri for the update. +1 (non-binding) on version 7.

          Show
          nroberts Nathan Roberts added a comment - Thanks Íñigo Goiri for the update. +1 (non-binding) on version 7.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch, Íñigo Goiri! Looks good overall, but I wonder about this change:

              // Get actual node physical resources
              int physicalMemoryMb = memoryMb;
              int physicalCores = virtualCores;
              ResourceCalculatorPlugin rcp =
                  ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf);
              if (rcp != null) {
                physicalMemoryMb = (int) (rcp.getPhysicalMemorySize() / (1024 * 1024));
                physicalCores = rcp.getNumProcessors();
              }
              this.physicalResource =
                  Resource.newInstance(physicalMemoryMb, physicalCores);
          

          This is not using the resource calculator the user may have configured for yarn.nodemanager.resource-calculator.class, and IIUC the value being reported here for total physical node resources will be compared against values being returned by that configured class. If the two classes don't match up then there could be some "interesting" and unexpected results. I think we need to be consistent and use the same class that's being used for resource monitoring.

          Rather than using read/write locks, we could avoid locks altogether by declaring the RMNodeImpl physicalResource field as volatile.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch, Íñigo Goiri ! Looks good overall, but I wonder about this change: // Get actual node physical resources int physicalMemoryMb = memoryMb; int physicalCores = virtualCores; ResourceCalculatorPlugin rcp = ResourceCalculatorPlugin.getResourceCalculatorPlugin( null , conf); if (rcp != null ) { physicalMemoryMb = ( int ) (rcp.getPhysicalMemorySize() / (1024 * 1024)); physicalCores = rcp.getNumProcessors(); } this .physicalResource = Resource.newInstance(physicalMemoryMb, physicalCores); This is not using the resource calculator the user may have configured for yarn.nodemanager.resource-calculator.class, and IIUC the value being reported here for total physical node resources will be compared against values being returned by that configured class. If the two classes don't match up then there could be some "interesting" and unexpected results. I think we need to be consistent and use the same class that's being used for resource monitoring. Rather than using read/write locks, we could avoid locks altogether by declaring the RMNodeImpl physicalResource field as volatile.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Thanks Jason Lowe for the review.

          Regarding the ResourceCalculatorPlugin, I'm very tempted to create a method ResourceCalculatorPlugin#getNMResourceCalculatorPlugin to encapsulate the code for RCP creation in ContainersMonitorImpl and make it easier to instantiate.

          Regarding the locks, the fewer locks the better, as it's just an assignment, volatile should be more than enough.

          Show
          elgoiri Íñigo Goiri added a comment - Thanks Jason Lowe for the review. Regarding the ResourceCalculatorPlugin , I'm very tempted to create a method ResourceCalculatorPlugin#getNMResourceCalculatorPlugin to encapsulate the code for RCP creation in ContainersMonitorImpl and make it easier to instantiate. Regarding the locks, the fewer locks the better, as it's just an assignment, volatile should be more than enough.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Setting ResourceCalculatorPlugin consistently for the NM and removing read/write lock.

          Show
          elgoiri Íñigo Goiri added a comment - Setting ResourceCalculatorPlugin consistently for the NM and removing read/write lock.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 15s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 6m 57s trunk passed
          +1 checkstyle 1m 30s trunk passed
          +1 mvnsite 2m 31s trunk passed
          +1 mvneclipse 1m 16s trunk passed
          +1 findbugs 4m 40s trunk passed
          +1 javadoc 1m 48s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          -1 mvninstall 0m 16s hadoop-yarn-common in the patch failed.
          -1 mvninstall 0m 20s hadoop-yarn-server-nodemanager in the patch failed.
          -1 compile 3m 20s root in the patch failed.
          -1 cc 3m 20s root in the patch failed.
          -1 javac 3m 20s root in the patch failed.
          -0 checkstyle 1m 29s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164)
          -1 mvnsite 0m 17s hadoop-yarn-common in the patch failed.
          -1 mvnsite 0m 20s hadoop-yarn-server-nodemanager in the patch failed.
          +1 mvneclipse 1m 27s the patch passed
          -1 whitespace 0m 0s The patch 1 line(s) with tabs.
          -1 findbugs 0m 17s hadoop-yarn-common in the patch failed.
          -1 findbugs 0m 20s hadoop-yarn-server-nodemanager in the patch failed.
          -1 javadoc 0m 17s hadoop-yarn-common in the patch failed.
          -1 unit 0m 17s hadoop-yarn-common in the patch failed.
          +1 unit 0m 31s hadoop-yarn-server-common in the patch passed.
          -1 unit 0m 22s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 41m 26s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 1m 0s hadoop-sls in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          109m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5356
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836870/YARN-5356.008.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 1cb8c794c790 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 352cbaa
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/whitespace-tabs.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13768/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13768/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 6m 57s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 2m 31s trunk passed +1 mvneclipse 1m 16s trunk passed +1 findbugs 4m 40s trunk passed +1 javadoc 1m 48s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch -1 mvninstall 0m 16s hadoop-yarn-common in the patch failed. -1 mvninstall 0m 20s hadoop-yarn-server-nodemanager in the patch failed. -1 compile 3m 20s root in the patch failed. -1 cc 3m 20s root in the patch failed. -1 javac 3m 20s root in the patch failed. -0 checkstyle 1m 29s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164) -1 mvnsite 0m 17s hadoop-yarn-common in the patch failed. -1 mvnsite 0m 20s hadoop-yarn-server-nodemanager in the patch failed. +1 mvneclipse 1m 27s the patch passed -1 whitespace 0m 0s The patch 1 line(s) with tabs. -1 findbugs 0m 17s hadoop-yarn-common in the patch failed. -1 findbugs 0m 20s hadoop-yarn-server-nodemanager in the patch failed. -1 javadoc 0m 17s hadoop-yarn-common in the patch failed. -1 unit 0m 17s hadoop-yarn-common in the patch failed. +1 unit 0m 31s hadoop-yarn-server-common in the patch passed. -1 unit 0m 22s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 41m 26s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 1m 0s hadoop-sls in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 109m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5356 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836870/YARN-5356.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 1cb8c794c790 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 352cbaa Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/whitespace-tabs.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13768/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13768/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13768/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 16s Maven dependency ordering for branch
          +1 mvninstall 7m 49s trunk passed
          +1 compile 8m 4s trunk passed
          +1 checkstyle 1m 33s trunk passed
          +1 mvnsite 2m 29s trunk passed
          +1 mvneclipse 1m 17s trunk passed
          +1 findbugs 4m 23s trunk passed
          +1 javadoc 2m 9s trunk passed
          0 mvndep 0m 19s Maven dependency ordering for patch
          -1 mvninstall 0m 27s hadoop-yarn-common in the patch failed.
          -1 mvninstall 0m 21s hadoop-yarn-server-nodemanager in the patch failed.
          -1 compile 3m 24s root in the patch failed.
          -1 cc 3m 24s root in the patch failed.
          -1 javac 3m 24s root in the patch failed.
          -0 checkstyle 1m 43s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164)
          -1 mvnsite 0m 29s hadoop-yarn-common in the patch failed.
          -1 mvnsite 0m 23s hadoop-yarn-server-nodemanager in the patch failed.
          +1 mvneclipse 1m 33s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 25s hadoop-yarn-common in the patch failed.
          -1 findbugs 0m 22s hadoop-yarn-server-nodemanager in the patch failed.
          +1 javadoc 2m 12s the patch passed
          -1 unit 0m 28s hadoop-yarn-common in the patch failed.
          +1 unit 0m 35s hadoop-yarn-server-common in the patch passed.
          -1 unit 0m 24s hadoop-yarn-server-nodemanager in the patch failed.
          -1 unit 37m 16s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 1m 1s hadoop-sls in the patch passed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          110m 41s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5356
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836918/YARN-5356.009.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 69e0299c39c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 20c4d8e
          Default Java 1.8.0_101
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/diff-checkstyle-root.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13771/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13771/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 7m 49s trunk passed +1 compile 8m 4s trunk passed +1 checkstyle 1m 33s trunk passed +1 mvnsite 2m 29s trunk passed +1 mvneclipse 1m 17s trunk passed +1 findbugs 4m 23s trunk passed +1 javadoc 2m 9s trunk passed 0 mvndep 0m 19s Maven dependency ordering for patch -1 mvninstall 0m 27s hadoop-yarn-common in the patch failed. -1 mvninstall 0m 21s hadoop-yarn-server-nodemanager in the patch failed. -1 compile 3m 24s root in the patch failed. -1 cc 3m 24s root in the patch failed. -1 javac 3m 24s root in the patch failed. -0 checkstyle 1m 43s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164) -1 mvnsite 0m 29s hadoop-yarn-common in the patch failed. -1 mvnsite 0m 23s hadoop-yarn-server-nodemanager in the patch failed. +1 mvneclipse 1m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 25s hadoop-yarn-common in the patch failed. -1 findbugs 0m 22s hadoop-yarn-server-nodemanager in the patch failed. +1 javadoc 2m 12s the patch passed -1 unit 0m 28s hadoop-yarn-common in the patch failed. +1 unit 0m 35s hadoop-yarn-server-common in the patch passed. -1 unit 0m 24s hadoop-yarn-server-nodemanager in the patch failed. -1 unit 37m 16s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 1s hadoop-sls in the patch passed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 110m 41s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5356 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836918/YARN-5356.009.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 69e0299c39c4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20c4d8e Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-compile-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13771/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13771/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13771/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Missing import.

          Show
          elgoiri Íñigo Goiri added a comment - Missing import.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 7m 52s trunk passed
          +1 compile 6m 56s trunk passed
          +1 checkstyle 1m 28s trunk passed
          +1 mvnsite 2m 24s trunk passed
          +1 mvneclipse 1m 18s trunk passed
          +1 findbugs 4m 23s trunk passed
          +1 javadoc 1m 58s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 23s the patch passed
          +1 compile 8m 1s the patch passed
          +1 cc 8m 1s the patch passed
          +1 javac 8m 1s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164)
          +1 mvnsite 2m 56s the patch passed
          +1 mvneclipse 1m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 0s the patch passed
          +1 javadoc 2m 6s the patch passed
          +1 unit 2m 28s hadoop-yarn-common in the patch passed.
          +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
          -1 unit 15m 39s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 38m 42s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 1m 9s hadoop-sls in the patch passed.
          +1 asflicense 0m 31s The patch does not generate ASF License warnings.
          133m 43s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue YARN-5356
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837145/YARN-5356.010.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 0c11cba8dd9d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0aafc12
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13786/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13786/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13786/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13786/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 7m 52s trunk passed +1 compile 6m 56s trunk passed +1 checkstyle 1m 28s trunk passed +1 mvnsite 2m 24s trunk passed +1 mvneclipse 1m 18s trunk passed +1 findbugs 4m 23s trunk passed +1 javadoc 1m 58s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 23s the patch passed +1 compile 8m 1s the patch passed +1 cc 8m 1s the patch passed +1 javac 8m 1s the patch passed -0 checkstyle 1m 38s root: The patch generated 4 new + 161 unchanged - 3 fixed = 165 total (was 164) +1 mvnsite 2m 56s the patch passed +1 mvneclipse 1m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 0s the patch passed +1 javadoc 2m 6s the patch passed +1 unit 2m 28s hadoop-yarn-common in the patch passed. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. -1 unit 15m 39s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 38m 42s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 1m 9s hadoop-sls in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 133m 43s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue YARN-5356 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837145/YARN-5356.010.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 0c11cba8dd9d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0aafc12 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13786/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13786/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13786/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13786/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          elgoiri Íñigo Goiri added a comment -

          The failed unit test doesn't seem related to the patch.

          Show
          elgoiri Íñigo Goiri added a comment - The failed unit test doesn't seem related to the patch.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch!

          I don't think getNMResourceCalculatorPlugin does quite the right thing for this case. Note that it tries to get the configured container monitor resource calculator with a fallback to the node monitor resource calculator. The resource monitor only uses the node monitor resource calculator, not the container monitor resource calculator. So theoretically we could be doing the wrong thing if the container monitor calculator did something different than the node monitor calculator. We need to use the same calculator that the NodeResourceMonitorImpl does, otherwise we can still have the mismatched calculators issue.

          We could add two utility methods for creating the calculator, one for the node monitor and one for the container monitor, and make sure we call the same one that NodeResourceMonitorImpl does.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! I don't think getNMResourceCalculatorPlugin does quite the right thing for this case. Note that it tries to get the configured container monitor resource calculator with a fallback to the node monitor resource calculator. The resource monitor only uses the node monitor resource calculator, not the container monitor resource calculator. So theoretically we could be doing the wrong thing if the container monitor calculator did something different than the node monitor calculator. We need to use the same calculator that the NodeResourceMonitorImpl does, otherwise we can still have the mismatched calculators issue. We could add two utility methods for creating the calculator, one for the node monitor and one for the container monitor, and make sure we call the same one that NodeResourceMonitorImpl does.
          Hide
          elgoiri Íñigo Goiri added a comment -

          Fixing ResourceCalculatorPlugin configs.

          Show
          elgoiri Íñigo Goiri added a comment - Fixing ResourceCalculatorPlugin configs.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 8m 28s trunk passed
          +1 compile 12m 7s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 3m 11s trunk passed
          +1 mvneclipse 2m 1s trunk passed
          +1 findbugs 4m 34s trunk passed
          +1 javadoc 2m 29s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 10s the patch passed
          +1 compile 10m 8s the patch passed
          +1 cc 10m 8s the patch passed
          +1 javac 10m 8s the patch passed
          -0 checkstyle 1m 44s root: The patch generated 4 new + 163 unchanged - 3 fixed = 167 total (was 166)
          +1 mvnsite 3m 33s the patch passed
          +1 mvneclipse 2m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 47s the patch passed
          +1 javadoc 2m 48s the patch passed
          +1 unit 2m 41s hadoop-yarn-common in the patch passed.
          +1 unit 0m 46s hadoop-yarn-server-common in the patch passed.
          -1 unit 16m 13s hadoop-yarn-server-nodemanager in the patch failed.
          -1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 1m 13s hadoop-sls in the patch passed.
          +1 asflicense 0m 54s The patch does not generate ASF License warnings.
          150m 12s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
            hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue YARN-5356
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837811/YARN-5356.011.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
          uname Linux 0aad50ac5955 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / acd509d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13810/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13810/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 8m 28s trunk passed +1 compile 12m 7s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 3m 11s trunk passed +1 mvneclipse 2m 1s trunk passed +1 findbugs 4m 34s trunk passed +1 javadoc 2m 29s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 10s the patch passed +1 compile 10m 8s the patch passed +1 cc 10m 8s the patch passed +1 javac 10m 8s the patch passed -0 checkstyle 1m 44s root: The patch generated 4 new + 163 unchanged - 3 fixed = 167 total (was 166) +1 mvnsite 3m 33s the patch passed +1 mvneclipse 2m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 47s the patch passed +1 javadoc 2m 48s the patch passed +1 unit 2m 41s hadoop-yarn-common in the patch passed. +1 unit 0m 46s hadoop-yarn-server-common in the patch passed. -1 unit 16m 13s hadoop-yarn-server-nodemanager in the patch failed. -1 unit 39m 54s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 1m 13s hadoop-sls in the patch passed. +1 asflicense 0m 54s The patch does not generate ASF License warnings. 150m 12s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager   hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-5356 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837811/YARN-5356.011.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 0aad50ac5955 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / acd509d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13810/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13810/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-tools/hadoop-sls U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13810/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch! The unit test failures appear to be unrelated. The TestQueuingContainerManager failure is tracked by YARN-5377. The TestAMRestart failure is tracked by YARN-5043. I filed YARN-5859 for the TestResourceLocalization failure.

          +1 for the latest patch. Committing this.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! The unit test failures appear to be unrelated. The TestQueuingContainerManager failure is tracked by YARN-5377 . The TestAMRestart failure is tracked by YARN-5043 . I filed YARN-5859 for the TestResourceLocalization failure. +1 for the latest patch. Committing this.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Íñigo Goiri for the contribution and to Nathan Roberts and Kuhu Shukla for additional review! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks to Íñigo Goiri for the contribution and to Nathan Roberts and Kuhu Shukla for additional review! I committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10790 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10790/)
          YARN-5356. NodeManager should communicate physical resource capability (jlowe: rev 3f93ac0733058238a2c8f23960c986c71dca0e02)

          • (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java
          • (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10790 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10790/ ) YARN-5356 . NodeManager should communicate physical resource capability (jlowe: rev 3f93ac0733058238a2c8f23960c986c71dca0e02) (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/RegisterNodeManagerRequestPBImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/TestYarnServerApiClasses.java (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java

            People

            • Assignee:
              elgoiri Íñigo Goiri
              Reporter:
              nroberts Nathan Roberts
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development