Hadoop YARN
  1. Hadoop YARN
  2. YARN-223

Change processTree interface to work better with native code

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Critical Critical
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 3.0.0, 2.0.3-alpha, trunk-win, 0.23.6
        • Component/s: None
        • Labels:
          None

          Description

          Problem is that on every update of processTree new object is required. This is undesired when working with processTree implementation in native code.

          replace ProcessTree.getProcessTree() with updateProcessTree(). No new object allocation is needed and it simplify application code a bit.

          1. Text File
            YARN-223-branch-trunk-win.1.patch
            31 kB
            Chris Nauroth
          2. Text File
            pstree-update6.txt
            27 kB
            Radim Kolar
          3. Text File
            pstree-update6.txt
            27 kB
            Radim Kolar
          4. Text File
            pstree-update4.txt
            27 kB
            Radim Kolar

            Issue Links

            is depended upon by

            Bug - A problem which impairs or prevents the functions of the product. YARN-234 YARN support for Windows process tree and resource calculation

            • Major - Major loss of function.
            • Resolved

              Activity

              Descending order - Click to sort in ascending order
              Hide
              Permalink
              Suresh Srinivas added a comment -

              Chris, thanks for the merged patch. It looks good. I will commit it to branch-trunk-win.

              Show
              Suresh Srinivas added a comment - Chris, thanks for the merged patch. It looks good. I will commit it to branch-trunk-win.
              Hide
              Permalink
              Chris Nauroth added a comment -

              branch-trunk-win diverged slightly due to committing YARN-234 ahead of this. We have a merge conflict now when trying to merge trunk to branch-trunk-win. I am attaching a separate patch for branch-trunk-win to resolve the merge conflict.

              Show
              Chris Nauroth added a comment - branch-trunk-win diverged slightly due to committing YARN-234 ahead of this. We have a merge conflict now when trying to merge trunk to branch-trunk-win. I am attaching a separate patch for branch-trunk-win to resolve the merge conflict.
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-trunk #1291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1291/)
              YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

              Result = SUCCESS
              llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1291 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1291/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/)
              YARN-223. Change processTree interface to work better with native code (Radim Kolar via tgraves) (Revision 1424590)

              Result = UNSTABLE
              tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424590
              Files :

              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
              • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #470 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/ ) YARN-223 . Change processTree interface to work better with native code (Radim Kolar via tgraves) (Revision 1424590) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424590 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Hide
              Permalink
              Luke Lu added a comment -

              I think the PreCommit test patch system should either handle cross project patches or outright reject such patches.

              Show
              Luke Lu added a comment - I think the PreCommit test patch system should either handle cross project patches or outright reject such patches.
              Hide
              Permalink
              Luke Lu added a comment -

              Sorry folks! I got too used to seeing mapreduce code in YARN

              Show
              Luke Lu added a comment - Sorry folks! I got too used to seeing mapreduce code in YARN
              Hide
              Permalink
              Thomas Graves added a comment -

              I pulled this into branch-0.23

              Show
              Thomas Graves added a comment - I pulled this into branch-0.23
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Hdfs-trunk #1260 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1260/)
              YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

              Result = FAILURE
              llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Show
              Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1260 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1260/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = FAILURE llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Hide
              Permalink
              Dennis Y added a comment -

              I've created https://issues.apache.org/jira/browse/MAPREDUCE-4895 and submitted patch

              Show
              Dennis Y added a comment - I've created https://issues.apache.org/jira/browse/MAPREDUCE-4895 and submitted patch
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Yarn-trunk #71 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/71/)
              YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

              Result = SUCCESS
              llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Show
              Hudson added a comment - Integrated in Hadoop-Yarn-trunk #71 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/71/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Hide
              Permalink
              Dennis Y added a comment -

              This patch breaks compilation of 'Apache Hadoop Gridmix' on branch-2.
              There is an import of org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.ProcResourceValues which were removed by this patch.

              2012-12-20 13:04:28.354 [INFO] ------------------------------------------------------------------------
              2012-12-20 13:04:28.356 [INFO] Building Apache Hadoop Gridmix 2.0.3-SNAPSHOT
              2012-12-20 13:04:28.357 [INFO] ------------------------------------------------------------------------
              2012-12-20 13:04:28.423 [INFO]
              2012-12-20 13:04:28.425 [INFO] — maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-gridmix —
              2012-12-20 13:04:28.426 [INFO]
              2012-12-20 13:04:28.428 [INFO] — maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-gridmix —
              2012-12-20 13:04:28.443 [INFO] Executing tasks
              2012-12-20 13:04:28.444
              2012-12-20 13:04:28.445 main:
              2012-12-20 13:04:28.446 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
              2012-12-20 13:04:28.448 [INFO] Executed tasks
              2012-12-20 13:04:28.450 [INFO]
              2012-12-20 13:04:28.452 [INFO] — maven-resources-plugin:2.2:resources (default-resources) @ hadoop-gridmix —
              2012-12-20 13:04:28.454 [INFO] Using default encoding to copy filtered resources.
              2012-12-20 13:04:28.455 [INFO]
              2012-12-20 13:04:28.457 [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-gridmix —
              2012-12-20 13:04:28.459 [INFO] Compiling 44 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/classes
              2012-12-20 13:04:28.638 [INFO]
              2012-12-20 13:04:28.640 [INFO] — maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-gridmix —
              2012-12-20 13:04:28.642 [INFO] Using default encoding to copy filtered resources.
              2012-12-20 13:04:28.643 [INFO]
              2012-12-20 13:04:28.645 [INFO] — maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-gridmix —
              2012-12-20 13:04:28.660 [INFO] Executing tasks
              2012-12-20 13:04:28.661
              2012-12-20 13:04:28.662 main:
              2012-12-20 13:04:28.663 [delete] Deleting directory /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
              2012-12-20 13:04:28.665 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir
              2012-12-20 13:04:28.667 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/log
              2012-12-20 13:04:28.669 [INFO] Executed tasks
              2012-12-20 13:04:28.670 [INFO]
              2012-12-20 13:04:28.672 [INFO] — maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-gridmix —
              2012-12-20 13:04:28.674 [INFO] Compiling 16 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-classes
              2012-12-20 13:04:28.774 [INFO] -------------------------------------------------------------
              2012-12-20 13:04:28.776 [ERROR] COMPILATION ERROR :
              2012-12-20 13:04:28.778 [INFO] -------------------------------------------------------------
              2012-12-20 13:04:28.780 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java:[35,59] cannot find symbol
              2012-12-20 13:04:28.781 symbol : class ProcResourceValues
              2012-12-20 13:04:28.782 location: class org.apache.hadoop.yarn.util.ResourceCalculatorPlugin
              2012-12-20 13:04:28.784 [INFO] 1 error
              2012-12-20 13:04:28.786 [INFO] -------------------------------------------------------------

              Show
              Dennis Y added a comment - This patch breaks compilation of 'Apache Hadoop Gridmix' on branch-2. There is an import of org.apache.hadoop.yarn.util.ResourceCalculatorPlugin.ProcResourceValues which were removed by this patch. 2012-12-20 13:04:28.354 [INFO] ------------------------------------------------------------------------ 2012-12-20 13:04:28.356 [INFO] Building Apache Hadoop Gridmix 2.0.3-SNAPSHOT 2012-12-20 13:04:28.357 [INFO] ------------------------------------------------------------------------ 2012-12-20 13:04:28.423 [INFO] 2012-12-20 13:04:28.425 [INFO] — maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-gridmix — 2012-12-20 13:04:28.426 [INFO] 2012-12-20 13:04:28.428 [INFO] — maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-gridmix — 2012-12-20 13:04:28.443 [INFO] Executing tasks 2012-12-20 13:04:28.444 2012-12-20 13:04:28.445 main: 2012-12-20 13:04:28.446 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.448 [INFO] Executed tasks 2012-12-20 13:04:28.450 [INFO] 2012-12-20 13:04:28.452 [INFO] — maven-resources-plugin:2.2:resources (default-resources) @ hadoop-gridmix — 2012-12-20 13:04:28.454 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.455 [INFO] 2012-12-20 13:04:28.457 [INFO] — maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-gridmix — 2012-12-20 13:04:28.459 [INFO] Compiling 44 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/classes 2012-12-20 13:04:28.638 [INFO] 2012-12-20 13:04:28.640 [INFO] — maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-gridmix — 2012-12-20 13:04:28.642 [INFO] Using default encoding to copy filtered resources. 2012-12-20 13:04:28.643 [INFO] 2012-12-20 13:04:28.645 [INFO] — maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-gridmix — 2012-12-20 13:04:28.660 [INFO] Executing tasks 2012-12-20 13:04:28.661 2012-12-20 13:04:28.662 main: 2012-12-20 13:04:28.663 [delete] Deleting directory /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.665 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-dir 2012-12-20 13:04:28.667 [mkdir] Created dir: /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/log 2012-12-20 13:04:28.669 [INFO] Executed tasks 2012-12-20 13:04:28.670 [INFO] 2012-12-20 13:04:28.672 [INFO] — maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-gridmix — 2012-12-20 13:04:28.674 [INFO] Compiling 16 source files to /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/target/test-classes 2012-12-20 13:04:28.774 [INFO] ------------------------------------------------------------- 2012-12-20 13:04:28.776 [ERROR] COMPILATION ERROR : 2012-12-20 13:04:28.778 [INFO] ------------------------------------------------------------- 2012-12-20 13:04:28.780 [ERROR] /home/jenkins/jenkins/workspace/Hadoop-common-Integration-branch-2-gd/yhadoop-common/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java: [35,59] cannot find symbol 2012-12-20 13:04:28.781 symbol : class ProcResourceValues 2012-12-20 13:04:28.782 location: class org.apache.hadoop.yarn.util.ResourceCalculatorPlugin 2012-12-20 13:04:28.784 [INFO] 1 error 2012-12-20 13:04:28.786 [INFO] -------------------------------------------------------------
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-trunk-Commit #3144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3144/)
              YARN-223. Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244)

              Result = SUCCESS
              llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244
              Files :

              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
              • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Show
              Hudson added a comment - Integrated in Hadoop-trunk-Commit #3144 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3144/ ) YARN-223 . Update process tree instead of getting new process trees. (Radim Kolar via llu) (Revision 1424244) Result = SUCCESS llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1424244 Files : /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
              Hide
              Permalink
              Luke Lu added a comment -

              Committed to trunk and branch-2. Thanks Radim for the patch and Chris and Bikas for the review.

              Show
              Luke Lu added a comment - Committed to trunk and branch-2. Thanks Radim for the patch and Chris and Bikas for the review.
              Hide
              Permalink
              Luke Lu added a comment -

              Upon second look, though the current ResourceCalculatorProcessTree interface is not ideal (checkPidPgrpidMatches threw me off, which should simply be checkOwnership), it's general enough I think.

              +1 for the patch. Will commit shortly.

              Show
              Luke Lu added a comment - Upon second look, though the current ResourceCalculatorProcessTree interface is not ideal (checkPidPgrpidMatches threw me off, which should simply be checkOwnership), it's general enough I think. +1 for the patch. Will commit shortly.
              Hide
              Permalink
              Radim Kolar added a comment -

              currently every system (windows and linux) which have resource calculator has process tree.

              if hypothetical system has just resource calculator but not resource process tree then it will miss just some stats currently sent to RM but not used anywhere. Most important function - watching container resources will be still missing without process tree.

              Show
              Radim Kolar added a comment - currently every system (windows and linux) which have resource calculator has process tree. if hypothetical system has just resource calculator but not resource process tree then it will miss just some stats currently sent to RM but not used anywhere. Most important function - watching container resources will be still missing without process tree.
              Hide
              Permalink
              Radim Kolar added a comment -

              process tree is standalone component for some time.

              Show
              Radim Kolar added a comment - process tree is standalone component for some time.
              Hide
              Permalink
              Luke Lu added a comment -
              1. The changes to eliminate usage of ResourceCalculatorPlugin is incorrect. Process tree is an impl detail of ResourceCalculatorPlugin. Many systems don't have process tree or equivalent at all.
              2. You only need to change the interface of ResourceCalculatorProcessTree and impl of ProcfsBasedProcessTree.
              Show
              Luke Lu added a comment - The changes to eliminate usage of ResourceCalculatorPlugin is incorrect. Process tree is an impl detail of ResourceCalculatorPlugin. Many systems don't have process tree or equivalent at all. You only need to change the interface of ResourceCalculatorProcessTree and impl of ProcfsBasedProcessTree.
              Hide
              Permalink
              Radim Kolar added a comment -

              i need to get this committed. Its one month already.

              Show
              Radim Kolar added a comment - i need to get this committed. Its one month already.
              Hide
              Permalink
              Radim Kolar added a comment -

              its private api

              @InterfaceAudience.Private
              @InterfaceStability.Unstable

              Show
              Radim Kolar added a comment - its private api @InterfaceAudience.Private @InterfaceStability.Unstable
              Hide
              Permalink
              Bikas Saha added a comment -

              +1 for the code and approach. The patch changes some public members. Not sure about those changes since they may not meet back-compat requirements.

              Show
              Bikas Saha added a comment - +1 for the code and approach. The patch changes some public members. Not sure about those changes since they may not meet back-compat requirements.
              Hide
              Permalink
              Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12560809/pstree-update6.txt
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 4 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-YARN-Build/223//testReport/
              Console output: https://builds.apache.org/job/PreCommit-YARN-Build/223//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12560809/pstree-update6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/223//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/223//console This message is automatically generated.
              Hide
              Permalink
              Radim Kolar added a comment -

              reupload patch to test QA bot

              Show
              Radim Kolar added a comment - reupload patch to test QA bot
              Hide
              Permalink
              Hadoop QA added a comment -

              +1 overall. Here are the results of testing the latest attachment
              http://issues.apache.org/jira/secure/attachment/12554602/pstree-update6.txt
              against trunk revision .

              +1 @author. The patch does not contain any @author tags.

              +1 tests included. The patch appears to include 4 new or modified test files.

              +1 javac. The applied patch does not increase the total number of javac compiler warnings.

              +1 javadoc. The javadoc tool did not generate any warning messages.

              +1 eclipse:eclipse. The patch built with eclipse:eclipse.

              +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

              +1 release audit. The applied patch does not increase the total number of release audit warnings.

              +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

              +1 contrib tests. The patch passed contrib unit tests.

              Test results: https://builds.apache.org/job/PreCommit-YARN-Build/171//testReport/
              Console output: https://builds.apache.org/job/PreCommit-YARN-Build/171//console

              This message is automatically generated.

              Show
              Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554602/pstree-update6.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/171//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/171//console This message is automatically generated.
              Hide
              Permalink
              Radim Kolar added a comment -

              will leave it as it is. version 6 passed QA bot checks.

              Show
              Radim Kolar added a comment - will leave it as it is. version 6 passed QA bot checks.
              Hide
              Permalink
              Radim Kolar added a comment -

              Just commit it before YARN-234 patch goes in. YARN-234 contains ResouceCalculator API change because of not understanding usage of getProcResourceValues. As this patch removes it, no changes are needed anymore.

              Show
              Radim Kolar added a comment - Just commit it before YARN-234 patch goes in. YARN-234 contains ResouceCalculator API change because of not understanding usage of getProcResourceValues. As this patch removes it, no changes are needed anymore.
              Hide
              Permalink
              Chris Nauroth added a comment -

              +1 patch looks good

              I successfully applied the patch to current trunk, compiled, and ran tests. A potential improvement is to add @Override annotations to the methods of TestResourceCalculatorProcessTree.EmptyProcessTree. This isn't directly related to your current change though, so I leave it up to you whether or not you want to update the patch.

              Show
              Chris Nauroth added a comment - +1 patch looks good I successfully applied the patch to current trunk, compiled, and ran tests. A potential improvement is to add @Override annotations to the methods of TestResourceCalculatorProcessTree.EmptyProcessTree. This isn't directly related to your current change though, so I leave it up to you whether or not you want to update the patch.
              Hide
              Permalink
              Radim Kolar added a comment -

              Migrated Task from ResourceCalculator to PsTree API.

              Show
              Radim Kolar added a comment - Migrated Task from ResourceCalculator to PsTree API.
              Hide
              Permalink
              Radim Kolar added a comment -

              Also remove getProcResourceValues from Resource Calculator API because it is not used anywhere and pluggable pstree has same functionality

              Show
              Radim Kolar added a comment - Also remove getProcResourceValues from Resource Calculator API because it is not used anywhere and pluggable pstree has same functionality
              Hide
              Permalink
              Radim Kolar added a comment -

              problems with old code: if you are using some system resources such as file handles in native library implementation of process tree then java can create kinda lot of objects of processTree type which are not garbage collected and temporary leaks system resources.

              If you start chaining references to original object, then you will have allocated system resources just once but you will create long chain on objects which could not be GC'd by java until all of them are unreferenced at end of container life.

              I tried both and didn't liked it, after inspection of code calling psTree it was discovered that simply updating object will be enough because code do not keeps old copy around while creating a new one.

              Show
              Radim Kolar added a comment - problems with old code: if you are using some system resources such as file handles in native library implementation of process tree then java can create kinda lot of objects of processTree type which are not garbage collected and temporary leaks system resources. If you start chaining references to original object, then you will have allocated system resources just once but you will create long chain on objects which could not be GC'd by java until all of them are unreferenced at end of container life. I tried both and didn't liked it, after inspection of code calling psTree it was discovered that simply updating object will be enough because code do not keeps old copy around while creating a new one.

                People

                • Assignee:
                  Radim Kolar
                  Reporter:
                  Radim Kolar
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  10 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development