Hadoop YARN
  1. Hadoop YARN
  2. YARN-223

Change processTree interface to work better with native code

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.0.3-alpha, trunk-win, 0.23.6
    • Component/s: None
    • Labels:
      None

      Description

      Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code.

      replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit.

      1. YARN-223-branch-trunk-win.1.patch
        31 kB
        Chris Nauroth
      2. pstree-update6.txt
        27 kB
        Radim Kolar
      3. pstree-update6.txt
        27 kB
        Radim Kolar
      4. pstree-update4.txt
        27 kB
        Radim Kolar

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          Chris, thanks for the merged patch. It looks good. I will commit it to branch-trunk-win.

          Show
          Suresh Srinivas added a comment - Chris, thanks for the merged patch. It looks good. I will commit it to branch-trunk-win.
          Hide
          Chris Nauroth added a comment -

          branch-trunk-win diverged slightly due to committing YARN-234 ahead of this. We have a merge conflict now when trying to merge trunk to branch-trunk-win. I am attaching a separate patch for branch-trunk-win to resolve the merge conflict.

          Show
          Chris Nauroth added a comment - branch-trunk-win diverged slightly due to committing YARN-234 ahead of this. We have a merge conflict now when trying to merge trunk to branch-trunk-win. I am attaching a separate patch for branch-trunk-win to resolve the merge conflict.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1291/)
          YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1291/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/)
          YARN-223. Change processTree interface to work better with native code (Radim Kolar via tgraves) (Revision 1424590)

          Result = UNSTABLE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424590
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/ ) YARN-223 . Change processTree interface to work better with native code (Radim Kolar via tgraves) (Revision 1424590) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424590 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Luke Lu added a comment -

          I think the PreCommit test patch system should either handle cross project patches or outright reject such patches.

          Show
          Luke Lu added a comment - I think the PreCommit test patch system should either handle cross project patches or outright reject such patches.
          Hide
          Luke Lu added a comment -

          Sorry folks! I got too used to seeing mapreduce code in YARN

          Show
          Luke Lu added a comment - Sorry folks! I got too used to seeing mapreduce code in YARN
          Hide
          Thomas Graves added a comment -

          I pulled this into branch-0.23

          Show
          Thomas Graves added a comment - I pulled this into branch-0.23
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1260 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1260/)
          YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

          Result = FAILURE
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1260 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1260/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Dennis Y added a comment -
          Show
          Dennis Y added a comment - I've created https://issues.apache.org/jira/browse/MAPREDUCE-4895 and submitted patch
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #71 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/71/)
          YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #71 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/71/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Dennis Y added a comment -

          This patch breaks compilation of 'Apache Hadoop Gridmix' on branch-2.
          There is an import of org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.ProcResourceValues which were removed by this patch.

          2012-12-20 13:04:28.354 [INFO] ------------------------------------------------------------------------
          2012-12-20 13:04:28.356 [INFO] Building Apache Hadoop Gridmix 2.0.3-SNAPSHOT
          2012-12-20 13:04:28.357 [INFO] ------------------------------------------------------------------------
          2012-12-20 13:04:28.423 [INFO]
          2012-12-20 13:04:28.425 [INFO] — maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-gridmix —
          2012-12-20 13:04:28.426 [INFO]
          2012-12-20 13:04:28.428 [INFO] — maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-gridmix —
          2012-12-20 13:04:28.443 [INFO] Executing tasks
          2012-12-20 13:04:28.444
          2012-12-20 13:04:28.445 main:
          2012-12-20 13:04:28.446 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
          2012-12-20 13:04:28.448 [INFO] Executed tasks
          2012-12-20 13:04:28.450 [INFO]
          2012-12-20 13:04:28.452 [INFO] — maven-resources-plugin:2.2:resources (default-resources) @ hadoop-gridmix —
          2012-12-20 13:04:28.454 [INFO] Using default encoding to copy filtered resources.
          2012-12-20 13:04:28.455 [INFO]
          2012-12-20 13:04:28.457 [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-gridmix —
          2012-12-20 13:04:28.459 [INFO] Compiling 44 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/classes
          2012-12-20 13:04:28.638 [INFO]
          2012-12-20 13:04:28.640 [INFO] — maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-gridmix —
          2012-12-20 13:04:28.642 [INFO] Using default encoding to copy filtered resources.
          2012-12-20 13:04:28.643 [INFO]
          2012-12-20 13:04:28.645 [INFO] — maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-gridmix —
          2012-12-20 13:04:28.660 [INFO] Executing tasks
          2012-12-20 13:04:28.661
          2012-12-20 13:04:28.662 main:
          2012-12-20 13:04:28.663 [delete] Deleting directory /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
          2012-12-20 13:04:28.665 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
          2012-12-20 13:04:28.667 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/log
          2012-12-20 13:04:28.669 [INFO] Executed tasks
          2012-12-20 13:04:28.670 [INFO]
          2012-12-20 13:04:28.672 [INFO] — maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-gridmix —
          2012-12-20 13:04:28.674 [INFO] Compiling 16 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-classes
          2012-12-20 13:04:28.774 [INFO] -------------------------------------------------------------
          2012-12-20 13:04:28.776 [ERROR] COMPILATION ERROR :
          2012-12-20 13:04:28.778 [INFO] -------------------------------------------------------------
          2012-12-20 13:04:28.780 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java:[35,59] cannot find symbol
          2012-12-20 13:04:28.781 symbol : class ProcResourceValues
          2012-12-20 13:04:28.782 location: class org.apache.hadoop.yarn.util.ResourceCalculatorPlugin
          2012-12-20 13:04:28.784 [INFO] 1 error
          2012-12-20 13:04:28.786 [INFO] -------------------------------------------------------------

          Show
          Dennis Y added a comment - This patch breaks compilation of 'Apache Hadoop Gridmix' on branch-2. There is an import of org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.ProcResourceValues which were removed by this patch. 2012-12-20 13:04:28.354 [INFO] ------------------------------------------------------------------------ 2012-12-20 13:04:28.356 [INFO] Building Apache Hadoop Gridmix 2.0.3-SNAPSHOT 2012-12-20 13:04:28.357 [INFO] ------------------------------------------------------------------------ 2012-12-20 13:04:28.423 [INFO] 2012-12-20 13:04:28.425 [INFO] — maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-gridmix — 2012-12-20 13:04:28.426 [INFO] 2012-12-20 13:04:28.428 [INFO] — maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-gridmix — 2012-12-20 13:04:28.443 [INFO] Executing tasks 2012-12-20 13:04:28.444 2012-12-20 13:04:28.445 main: 2012-12-20 13:04:28.446 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.448 [INFO] Executed tasks 2012-12-20 13:04:28.450 [INFO] 2012-12-20 13:04:28.452 [INFO] — maven-resources-plugin:2.2:resources (default-resources) @ hadoop-gridmix — 2012-12-20 13:04:28.454 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.455 [INFO] 2012-12-20 13:04:28.457 [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-gridmix — 2012-12-20 13:04:28.459 [INFO] Compiling 44 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/classes 2012-12-20 13:04:28.638 [INFO] 2012-12-20 13:04:28.640 [INFO] — maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-gridmix — 2012-12-20 13:04:28.642 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.643 [INFO] 2012-12-20 13:04:28.645 [INFO] — maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-gridmix — 2012-12-20 13:04:28.660 [INFO] Executing tasks 2012-12-20 13:04:28.661 2012-12-20 13:04:28.662 main: 2012-12-20 13:04:28.663 [delete] Deleting directory /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.665 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.667 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/log 2012-12-20 13:04:28.669 [INFO] Executed tasks 2012-12-20 13:04:28.670 [INFO] 2012-12-20 13:04:28.672 [INFO] — maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-gridmix — 2012-12-20 13:04:28.674 [INFO] Compiling 16 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-classes 2012-12-20 13:04:28.774 [INFO] ------------------------------------------------------------- 2012-12-20 13:04:28.776 [ERROR] COMPILATION ERROR : 2012-12-20 13:04:28.778 [INFO] ------------------------------------------------------------- 2012-12-20 13:04:28.780 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java: [35,59] cannot find symbol 2012-12-20 13:04:28.781 symbol : class ProcResourceValues 2012-12-20 13:04:28.782 location: class org.apache.hadoop.yarn.util.ResourceCalculatorPlugin 2012-12-20 13:04:28.784 [INFO] 1 error 2012-12-20 13:04:28.786 [INFO] -------------------------------------------------------------
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3144/)
          YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

          Result = SUCCESS
          llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3144/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Luke Lu added a comment -

          Committed to trunk and branch-2. Thanks Radim for the patch and Chris and Bikas for the review.

          Show
          Luke Lu added a comment - Committed to trunk and branch-2. Thanks Radim for the patch and Chris and Bikas for the review.
          Hide
          Luke Lu added a comment -

          Upon second look, though the current ResourceCalculatorProcessTree interface is not ideal (checkPidPgrpidMatches threw me off, which should simply be checkOwnership), it's general enough I think.

          +1 for the patch. Will commit shortly.

          Show
          Luke Lu added a comment - Upon second look, though the current ResourceCalculatorProcessTree interface is not ideal (checkPidPgrpidMatches threw me off, which should simply be checkOwnership), it's general enough I think. +1 for the patch. Will commit shortly.
          Hide
          Radim Kolar added a comment -

          currently every system (windows and linux) which have resource calculator has process tree.

          if hypothetical system has just resource calculator but not resource process tree then it will miss just some stats currently sent to RM but not used anywhere. Most important function - watching container resources will be still missing without process tree.

          Show
          Radim Kolar added a comment - currently every system (windows and linux) which have resource calculator has process tree. if hypothetical system has just resource calculator but not resource process tree then it will miss just some stats currently sent to RM but not used anywhere. Most important function - watching container resources will be still missing without process tree.
          Hide
          Radim Kolar added a comment -

          process tree is standalone component for some time.

          Show
          Radim Kolar added a comment - process tree is standalone component for some time.
          Hide
          Luke Lu added a comment -
          1. The changes to eliminate usage of ResourceCalculatorPlugin is incorrect. Process tree is an impl detail of ResourceCalculatorPlugin. Many systems don't have process tree or equivalent at all.
          2. You only need to change the interface of ResourceCalculatorProcessTree and impl of ProcfsBasedProcessTree.
          Show
          Luke Lu added a comment - The changes to eliminate usage of ResourceCalculatorPlugin is incorrect. Process tree is an impl detail of ResourceCalculatorPlugin. Many systems don't have process tree or equivalent at all. You only need to change the interface of ResourceCalculatorProcessTree and impl of ProcfsBasedProcessTree.
          Hide
          Radim Kolar added a comment -

          i need to get this committed. Its one month already.

          Show
          Radim Kolar added a comment - i need to get this committed. Its one month already.
          Hide
          Radim Kolar added a comment -

          its private api

          @InterfaceAudience.Private
          @InterfaceStability.Unstable

          Show
          Radim Kolar added a comment - its private api @InterfaceAudience.Private @InterfaceStability.Unstable
          Hide
          Bikas Saha added a comment -

          +1 for the code and approach. The patch changes some public members. Not sure about those changes since they may not meet back-compat requirements.

          Show
          Bikas Saha added a comment - +1 for the code and approach. The patch changes some public members. Not sure about those changes since they may not meet back-compat requirements.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12560809/pstree-update6.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/223//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/223//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12560809/pstree-update6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/223//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/223//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          reupload patch to test QA bot

          Show
          Radim Kolar added a comment - reupload patch to test QA bot
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554602/pstree-update6.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/171//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/171//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554602/pstree-update6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/171//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/171//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          will leave it as it is. version 6 passed QA bot checks.

          Show
          Radim Kolar added a comment - will leave it as it is. version 6 passed QA bot checks.
          Hide
          Radim Kolar added a comment -

          Just commit it before YARN-234 patch goes in. YARN-234 contains ResouceCalculator API change because of not understanding usage of getProcResourceValues. As this patch removes it, no changes are needed anymore.

          Show
          Radim Kolar added a comment - Just commit it before YARN-234 patch goes in. YARN-234 contains ResouceCalculator API change because of not understanding usage of getProcResourceValues. As this patch removes it, no changes are needed anymore.
          Hide
          Chris Nauroth added a comment -

          +1 patch looks good

          I successfully applied the patch to current trunk, compiled, and ran tests. A potential improvement is to add @Override annotations to the methods of TestResourceCalculatorProcessTree.EmptyProcessTree. This isn't directly related to your current change though, so I leave it up to you whether or not you want to update the patch.

          Show
          Chris Nauroth added a comment - +1 patch looks good I successfully applied the patch to current trunk, compiled, and ran tests. A potential improvement is to add @Override annotations to the methods of TestResourceCalculatorProcessTree.EmptyProcessTree. This isn't directly related to your current change though, so I leave it up to you whether or not you want to update the patch.
          Hide
          Radim Kolar added a comment -

          Migrated Task from ResourceCalculator to PsTree API.

          Show
          Radim Kolar added a comment - Migrated Task from ResourceCalculator to PsTree API.
          Hide
          Radim Kolar added a comment -

          Also remove getProcResourceValues from Resource Calculator API because it is not used anywhere and pluggable pstree has same functionality

          Show
          Radim Kolar added a comment - Also remove getProcResourceValues from Resource Calculator API because it is not used anywhere and pluggable pstree has same functionality
          Hide
          Radim Kolar added a comment -

          problems with old code: if you are using some system resources such as file handles in native library implementation of process tree then java can create kinda lot of objects of processTree type which are not garbage collected and temporary leaks system resources.

          If you start chaining references to original object, then you will have allocated system resources just once but you will create long chain on objects which could not be GC'd by java until all of them are unreferenced at end of container life.

          I tried both and didn't liked it, after inspection of code calling psTree it was discovered that simply updating object will be enough because code do not keeps old copy around while creating a new one.

          Show
          Radim Kolar added a comment - problems with old code: if you are using some system resources such as file handles in native library implementation of process tree then java can create kinda lot of objects of processTree type which are not garbage collected and temporary leaks system resources. If you start chaining references to original object, then you will have allocated system resources just once but you will create long chain on objects which could not be GC'd by java until all of them are unreferenced at end of container life. I tried both and didn't liked it, after inspection of code calling psTree it was discovered that simply updating object will be enough because code do not keeps old copy around while creating a new one.

            People

            • Assignee:
              Radim Kolar
              Reporter:
              Radim Kolar
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development