Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.4, 3.0.0, 2.0.3-alpha
    • Component/s: nodemanager
    • Labels:
      None
    • Environment:

      FreeBSD 64 bit

      Description

      Trunk version of Pluggable process tree. Work based on MAPREDUCE-4204

      1. plugable-pstree.txt
        42 kB
        Radim Kolar
      2. plugable-pstree-1.txt
        42 kB
        Radim Kolar
      3. plugable-pstree-2.txt
        42 kB
        Radim Kolar
      4. plugable-pstree-3.txt
        27 kB
        Radim Kolar
      5. plugable-pstree-4.txt
        28 kB
        Radim Kolar
      6. plugable-pstree-4-with-whitespace.txt
        46 kB
        Radim Kolar
      7. plugable-pstree-5-with-whitespace.txt
        46 kB
        Radim Kolar
      8. plugable-pstree-6-typofix.txt
        46 kB
        Radim Kolar
      9. plugable-pstree-trunk-merge-7.txt
        46 kB
        Radim Kolar
      10. plugable-pstree-trunk-merge-7.txt
        46 kB
        Radim Kolar
      11. plugable-pstree-trunk-merge-8.txt
        46 kB
        Radim Kolar
      12. pstree-0.23-merge.txt
        40 kB
        Radim Kolar
      13. pstree-0.23-merge-2.txt
        46 kB
        Radim Kolar

        Issue Links

          Activity

          Radim Kolar created issue -
          Radim Kolar made changes -
          Field Original Value New Value
          Attachment plugable-pstree.txt [ 12528392 ]
          Radim Kolar made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to MAPREDUCE-4204 [ MAPREDUCE-4204 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528392/plugable-pstree.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528392/plugable-pstree.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//artifact/trunk/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2404//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          Fixed bug in detection of pluggable process tree availability

          Show
          Radim Kolar added a comment - Fixed bug in detection of pluggable process tree availability
          Radim Kolar made changes -
          Attachment plugable-pstree-1.txt [ 12528407 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12528407/plugable-pstree-1.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2405//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2405//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528407/plugable-pstree-1.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2405//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2405//console This message is automatically generated.
          Arun C Murthy made changes -
          Fix Version/s 0.23.0 [ 12315570 ]
          Fix Version/s 2.0.0-alpha [ 12320354 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Hide
          Radim Kolar added a comment -

          I really need this to get commited.

          Show
          Radim Kolar added a comment - I really need this to get commited.
          Hide
          Bikas Saha added a comment -

          Thanks for the patch and sorry for not getting to it earlier.
          In future, could you please use the --no-prefix tag so that the diff does not have a/hadoop-mapreduce..../.. and is easier to apply.

          1) In general there are a bunch of extra spaces all over the place that would be good to remove.
          2) The comment looks like it has a typo in ResourceCalculatorPlugin.java

            /**
             * Get the ResourceCalculatorProcessTree configure it. This method will try
             * and return a ResourceCalculatorProcessTree available for this system.
          

          3. ContainerMonitorImpl.java
          There is an indentation issue in the first if. The second if statement is leaking Linux semantics into the code by trying to get process tree for pid 1. IMO, the fact that resourceCalculatorPlugin exists should be enough to ensure that underlying interfaces have been correctly implemented.

          	if (resourceCalculatorPlugin == null) {
          	        LOG.info("ResourceCalculatorPlugin is unavaiable on this system. "
          	            + this.getClass().getName() + " is disabled.");
          	        return false;
          	}
          
              if (resourceCalculatorPlugin.getResourceCalculatorProcessTree("1") == null ) {
                LOG.info("ProcessTree implementation is missing on this system. "
                    + this.getClass().getName() + " is disabled.");
                return false;
              }
          

          After removal of the following code can ContainerExecutor.isSetSidAvailable be removed too? Any more cleanup possible?

                          ProcfsBasedProcessTree pt =
                              new ProcfsBasedProcessTree(pId,
                                  ContainerExecutor.isSetsidAvailable);
          

          Other than these, it look good.

          Show
          Bikas Saha added a comment - Thanks for the patch and sorry for not getting to it earlier. In future, could you please use the --no-prefix tag so that the diff does not have a/hadoop-mapreduce..../.. and is easier to apply. 1) In general there are a bunch of extra spaces all over the place that would be good to remove. 2) The comment looks like it has a typo in ResourceCalculatorPlugin.java /** * Get the ResourceCalculatorProcessTree configure it. This method will try * and return a ResourceCalculatorProcessTree available for this system. 3. ContainerMonitorImpl.java There is an indentation issue in the first if. The second if statement is leaking Linux semantics into the code by trying to get process tree for pid 1. IMO, the fact that resourceCalculatorPlugin exists should be enough to ensure that underlying interfaces have been correctly implemented. if (resourceCalculatorPlugin == null ) { LOG.info( "ResourceCalculatorPlugin is unavaiable on this system. " + this .getClass().getName() + " is disabled." ); return false ; } if (resourceCalculatorPlugin.getResourceCalculatorProcessTree( "1" ) == null ) { LOG.info( "ProcessTree implementation is missing on this system. " + this .getClass().getName() + " is disabled." ); return false ; } After removal of the following code can ContainerExecutor.isSetSidAvailable be removed too? Any more cleanup possible? ProcfsBasedProcessTree pt = new ProcfsBasedProcessTree(pId, ContainerExecutor.isSetsidAvailable); Other than these, it look good.
          Hide
          Radim Kolar added a comment -

          Typo in comment, extra check for process number "1" removed.

          Show
          Radim Kolar added a comment - Typo in comment, extra check for process number "1" removed.
          Radim Kolar made changes -
          Attachment plugable-pstree-2.txt [ 12537833 ]
          Hide
          Radim Kolar added a comment -

          check for ContainerExecutor.isSetSidAvailable has nothing to do with pstree.

          ContainerExecutor needs to be redesigned to make it abstract with almost no code inside and then create UnixContainerExecuter which will do setsid check and other unix specific stuff moved there. If linux still needs some special features then it can subclass unixcontainerexecutor.

          Show
          Radim Kolar added a comment - check for ContainerExecutor.isSetSidAvailable has nothing to do with pstree. ContainerExecutor needs to be redesigned to make it abstract with almost no code inside and then create UnixContainerExecuter which will do setsid check and other unix specific stuff moved there. If linux still needs some special features then it can subclass unixcontainerexecutor.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537833/plugable-pstree-2.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2657//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2657//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537833/plugable-pstree-2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2657//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2657//console This message is automatically generated.
          Hide
          Bikas Saha added a comment -

          1) This code is mainly used to control the monitoring thread. The monitoring thread relies on availability of ResourceCalculatorProcessTree and not on resourceCalculatorPlugin. Hence, I would tend to replace ProcfsBasedProcessTree.isAvailable() with ResourceCalculatorProcessTree.isAvailable(). That is the approach taken in MAPREDUCE-4204.
          ResourceCalculatorPlugin is more of a system resource calculator for the machine resources. Getting ResourceCalculatorProcessTree impl from ResourceCalculatorPlugin formally connects them in the object model. Also, it does not sound like a standard pattern to have Class B become the factory of Class A.

             private boolean isEnabled() {
          -    if (!ProcfsBasedProcessTree.isAvailable()) {
          -      LOG.info("ProcessTree implementation is missing on this system. "
          -          + this.getClass().getName() + " is disabled.");
          -      return false;
          +    if (resourceCalculatorPlugin == null) {
          +            LOG.info("ResourceCalculatorPlugin is unavaiable on this system. "
          +                + this.getClass().getName() + " is disabled.");
          +            return false;
               }
          

          2. There are a bunch of diff's like the following that are adding noise to the patch and making it larger than it should be. Mind checking and cleaning up?

             /**
              * Checks if the ProcfsBasedProcessTree is available on this system.
          -   * 
          +   *
              * @return true if ProcfsBasedProcessTree is available. False otherwise.
              */
          
          Show
          Bikas Saha added a comment - 1) This code is mainly used to control the monitoring thread. The monitoring thread relies on availability of ResourceCalculatorProcessTree and not on resourceCalculatorPlugin. Hence, I would tend to replace ProcfsBasedProcessTree.isAvailable() with ResourceCalculatorProcessTree.isAvailable(). That is the approach taken in MAPREDUCE-4204 . ResourceCalculatorPlugin is more of a system resource calculator for the machine resources. Getting ResourceCalculatorProcessTree impl from ResourceCalculatorPlugin formally connects them in the object model. Also, it does not sound like a standard pattern to have Class B become the factory of Class A. private boolean isEnabled() { - if (!ProcfsBasedProcessTree.isAvailable()) { - LOG.info( "ProcessTree implementation is missing on this system. " - + this .getClass().getName() + " is disabled." ); - return false ; + if (resourceCalculatorPlugin == null ) { + LOG.info( "ResourceCalculatorPlugin is unavaiable on this system. " + + this .getClass().getName() + " is disabled." ); + return false ; } 2. There are a bunch of diff's like the following that are adding noise to the patch and making it larger than it should be. Mind checking and cleaning up? /** * Checks if the ProcfsBasedProcessTree is available on this system. - * + * * @ return true if ProcfsBasedProcessTree is available. False otherwise. */
          Hide
          Radim Kolar added a comment -

          static isAvailable() is worst possible design, because 1. its static and 2. its heavyweight compared to not null pointer check. Best design would be to have one factory per operation system supported which will create different OS specific objects - currently there are two: ResourceCalculatorProcessTree and ResourceCalculatorPlugin.

          ContainersMonitorImpl is using both ResourceCalculatorProcessTree and ResourceCalculatorProcessTree.

          these space issues are caused by editor killing trailing whitespace, which is good thing. Probably --ignore-whitespace in patch will strip it.

          But because i am really in hurry and this issue takes MONTHS to resolve for no apparent reason, i will code whatever design you want. If you want static isAvailable(), i will deliver it. Just state clearly what you want. I will throw out this code after few months, it will be replaced by pluggable resources, which are way better design.

          Show
          Radim Kolar added a comment - static isAvailable() is worst possible design, because 1. its static and 2. its heavyweight compared to not null pointer check. Best design would be to have one factory per operation system supported which will create different OS specific objects - currently there are two: ResourceCalculatorProcessTree and ResourceCalculatorPlugin. ContainersMonitorImpl is using both ResourceCalculatorProcessTree and ResourceCalculatorProcessTree. these space issues are caused by editor killing trailing whitespace, which is good thing. Probably --ignore-whitespace in patch will strip it. But because i am really in hurry and this issue takes MONTHS to resolve for no apparent reason, i will code whatever design you want. If you want static isAvailable(), i will deliver it. Just state clearly what you want. I will throw out this code after few months, it will be replaced by pluggable resources, which are way better design.
          Hide
          Bikas Saha added a comment -

          I am ok with the null pointer check. You are right that there should be a single factory getting OS specific resource objects. How about moving the procfsbasedprocesstree factory to resourcecalculatorprocesstree for now so that the code flow stays similar to what it currently is. Looks like you have some ideas for re-organizing this part of the code better. we could handle the factory creation and other refactorings etc as part of that change. sounds like a plan?

          Show
          Bikas Saha added a comment - I am ok with the null pointer check. You are right that there should be a single factory getting OS specific resource objects. How about moving the procfsbasedprocesstree factory to resourcecalculatorprocesstree for now so that the code flow stays similar to what it currently is. Looks like you have some ideas for re-organizing this part of the code better. we could handle the factory creation and other refactorings etc as part of that change. sounds like a plan?
          Hide
          Radim Kolar added a comment -

          Do not create processtree instance from resourcecalculator plugin. Make them separated.

          Show
          Radim Kolar added a comment - Do not create processtree instance from resourcecalculator plugin. Make them separated.
          Radim Kolar made changes -
          Attachment plugable-pstree-3.txt [ 12539020 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539020/plugable-pstree-3.txt
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2702//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539020/plugable-pstree-3.txt against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2702//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          check if ProcessTree is available before enabling monitoring

          Show
          Radim Kolar added a comment - check if ProcessTree is available before enabling monitoring
          Radim Kolar made changes -
          Attachment plugable-pstree-4.txt [ 12539022 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539022/plugable-pstree-4.txt
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2703//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539022/plugable-pstree-4.txt against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2703//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          now without removed whitespace lines

          Show
          Radim Kolar added a comment - now without removed whitespace lines
          Radim Kolar made changes -
          Attachment plugable-pstree-4-with-whitespace.txt [ 12539036 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539036/plugable-pstree-4-with-whitespace.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

          org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
          org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
          org.apache.hadoop.yarn.server.nodemanager.TestEventFlow
          org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
          org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
          org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2704//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2704//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539036/plugable-pstree-4-with-whitespace.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor org.apache.hadoop.yarn.server.nodemanager.TestEventFlow org.apache.hadoop.yarn.server.nodemanager.containermanager.TestContainerManager org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2704//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2704//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          avoid null pointer dereference in init()

          Show
          Radim Kolar added a comment - avoid null pointer dereference in init()
          Radim Kolar made changes -
          Attachment plugable-pstree-5-with-whitespace.txt [ 12539050 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539050/plugable-pstree-5-with-whitespace.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2705//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2705//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539050/plugable-pstree-5-with-whitespace.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2705//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2705//console This message is automatically generated.
          Hide
          Bikas Saha added a comment -

          Thanks for incorporating my comments. +1.

          Minor typo in unavailable

          +    if (resourceCalculatorPlugin == null) {
          +            LOG.info("ResourceCalculatorPlugin is unavaiable on this system. "
          +                + this.getClass().getName() + " is disabled.");
          +            return false;
          +    }
          +    if (ResourceCalculatorProcessTree.getResourceCalculatorProcessTree("0", processTreeClass, conf) == null) {
          +        LOG.info("ResourceCalculatorProcessTree is unavaiable on this system. "
          +                + this.getClass().getName() + " is disabled.");
          
          Show
          Bikas Saha added a comment - Thanks for incorporating my comments. +1. Minor typo in unavailable + if (resourceCalculatorPlugin == null ) { + LOG.info( "ResourceCalculatorPlugin is unavaiable on this system. " + + this .getClass().getName() + " is disabled." ); + return false ; + } + if (ResourceCalculatorProcessTree.getResourceCalculatorProcessTree( "0" , processTreeClass, conf) == null ) { + LOG.info( "ResourceCalculatorProcessTree is unavaiable on this system. " + + this .getClass().getName() + " is disabled." );
          Hide
          Radim Kolar added a comment -

          typo fixed

          Show
          Radim Kolar added a comment - typo fixed
          Radim Kolar made changes -
          Attachment plugable-pstree-6-typofix.txt [ 12539110 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539110/plugable-pstree-6-typofix.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2708//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2708//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539110/plugable-pstree-6-typofix.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2708//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2708//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          can be this commited?

          Show
          Radim Kolar added a comment - can be this commited?
          Arun C Murthy made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Key MAPREDUCE-4275 YARN-57
          Affects Version/s 3.0.0 [ 12320355 ]
          Component/s nodemanager [ 12319323 ]
          Component/s nodemanager [ 12315341 ]
          Tags yarn
          Arun C Murthy made changes -
          Assignee Radim Kolar [ hsn ]
          Hide
          Arun C Murthy added a comment -

          Radim, the patch looks right.

          Can you please rebase against trunk for one last time and I'll commit? Tx!

          Show
          Arun C Murthy added a comment - Radim, the patch looks right. Can you please rebase against trunk for one last time and I'll commit? Tx!
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Radim Kolar added a comment -

          merge with trunk

          Show
          Radim Kolar added a comment - merge with trunk
          Radim Kolar made changes -
          Attachment plugable-pstree-trunk-merge-7.txt [ 12543291 ]
          Hide
          Arun C Murthy added a comment -

          Submitting patch for hudson, I'll commit after. Thanks Radim.

          Show
          Arun C Murthy added a comment - Submitting patch for hudson, I'll commit after. Thanks Radim.
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Radim Kolar added a comment -

          reupload to wake QA bot.

          Show
          Radim Kolar added a comment - reupload to wake QA bot.
          Radim Kolar made changes -
          Attachment plugable-pstree-trunk-merge-7.txt [ 12543655 ]
          Show
          Radim Kolar added a comment - QA bot is dead https://builds.apache.org/view/Hadoop/job/PreCommit-YARN-Build/
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Just kicked Jenkins manually..

          Show
          Vinod Kumar Vavilapalli added a comment - Just kicked Jenkins manually..
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543655/plugable-pstree-trunk-merge-7.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/16//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543655/plugable-pstree-trunk-merge-7.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/16//console This message is automatically generated.
          Hide
          Radim Kolar added a comment -

          where is log with javac errors?

          Show
          Radim Kolar added a comment - where is log with javac errors?
          Hide
          Radim Kolar added a comment -

          one file was missing from patch because git do not tracks directory renames

          Show
          Radim Kolar added a comment - one file was missing from patch because git do not tracks directory renames
          Radim Kolar made changes -
          Attachment plugable-pstree-trunk-merge-8.txt [ 12543840 ]
          Hide
          Arun C Murthy added a comment -
          Show
          Arun C Murthy added a comment - Kicked hudson again... https://builds.apache.org/job/PreCommit-YARN-Build/33/
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543840/plugable-pstree-trunk-merge-8.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/33//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/33//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543840/plugable-pstree-trunk-merge-8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/33//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/33//console This message is automatically generated.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Radim!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Radim!
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12323268 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2705 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2705/)
          YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2705 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2705/ ) YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2768 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2768/)
          YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2768 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2768/ ) YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2729 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2729/)
          YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072)

          Result = FAILURE
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2729 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2729/ ) YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/)
          YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072)

          Result = FAILURE
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1159 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1159/ ) YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/)
          YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072)

          Result = SUCCESS
          acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1190 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1190/ ) YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1382072) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1382072 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Hide
          Radim Kolar added a comment -

          I need merge to 0.23, attaching patch with conflict resolved

          Show
          Radim Kolar added a comment - I need merge to 0.23, attaching patch with conflict resolved
          Radim Kolar made changes -
          Attachment pstree-0.23-merge.txt [ 12544453 ]
          Hide
          Robert Joseph Evans added a comment -

          The 0.23 patch does not compile. It looks like it is missing some new files that were added in as part of the original patch.

          Also before it can go into branch-0.23 it needs to go into branch-2 first. I have not had a lot of time to look at this in depth. It would be nice if Arun could weigh in on this JIRA going into branch-2 as he is the one who committed this originally, he is the branch-2 release manager and I think he has more experience with this code change then I do.

          Show
          Robert Joseph Evans added a comment - The 0.23 patch does not compile. It looks like it is missing some new files that were added in as part of the original patch. Also before it can go into branch-0.23 it needs to go into branch-2 first. I have not had a lot of time to look at this in depth. It would be nice if Arun could weigh in on this JIRA going into branch-2 as he is the one who committed this originally, he is the branch-2 release manager and I think he has more experience with this code change then I do.
          Hide
          Robert Joseph Evans added a comment -

          Someone also mentioned to me that this is a new feature which is kind of the opposite of what we are trying to do on 0.23 right now. But it looks like it is backwards compatible and a fairly small feature at that so if you really need it for 0.23 I think it should be OK but I do want to have more eyes look at it and it would be nice to get some test results on a secure cluster before putting it in.

          Show
          Robert Joseph Evans added a comment - Someone also mentioned to me that this is a new feature which is kind of the opposite of what we are trying to do on 0.23 right now. But it looks like it is backwards compatible and a fairly small feature at that so if you really need it for 0.23 I think it should be OK but I do want to have more eyes look at it and it would be nice to get some test results on a secure cluster before putting it in.
          Hide
          Radim Kolar added a comment -

          patch for 0.23 regenerated

          Show
          Radim Kolar added a comment - patch for 0.23 regenerated
          Radim Kolar made changes -
          Attachment pstree-0.23-merge-2.txt [ 12544824 ]
          Hide
          Robert Joseph Evans added a comment -

          I looked over the patch and it looks OK to me. It is a clean port of the original patch. I am +1 with this going into 0.23.4 assuming that Arun is OK with it going into branch-2. Because he checked this into trunk and not branch-2 I want to know for sure what his reasons where before I go checking this into branch-2 and branch-0.23.

          Show
          Robert Joseph Evans added a comment - I looked over the patch and it looks OK to me. It is a clean port of the original patch. I am +1 with this going into 0.23.4 assuming that Arun is OK with it going into branch-2. Because he checked this into trunk and not branch-2 I want to know for sure what his reasons where before I go checking this into branch-2 and branch-0.23.
          Hide
          Radim Kolar added a comment -

          Did you come to some conclusion about branch-0.23 merge, Arun & Evans?

          Show
          Radim Kolar added a comment - Did you come to some conclusion about branch-0.23 merge, Arun & Evans?
          Hide
          Robert Joseph Evans added a comment -

          Like I said before I am OK with letting it in, assuming that Arun is OK with it. I have not pushed for an answer from him on it.

          Show
          Robert Joseph Evans added a comment - Like I said before I am OK with letting it in, assuming that Arun is OK with it. I have not pushed for an answer from him on it.
          Hide
          Arun C Murthy added a comment -

          Bobby - I'm +1 on this. However, you make the final call since you RM releases from that branch. Thanks.

          Show
          Arun C Murthy added a comment - Bobby - I'm +1 on this. However, you make the final call since you RM releases from that branch. Thanks.
          Hide
          Radim Kolar added a comment -

          Fine. You both agree, then merge it already.

          Show
          Radim Kolar added a comment - Fine. You both agree, then merge it already.
          Hide
          Robert Joseph Evans added a comment -

          Sorry, you have every right to be frustrated. I have been distracted by a lot of stuff happening lately, and this fell off my radar. Thanks for reminding me. Merging this now...

          Show
          Robert Joseph Evans added a comment - Sorry, you have every right to be frustrated. I have been distracted by a lot of stuff happening lately, and this fell off my radar. Thanks for reminding me. Merging this now...
          Hide
          Robert Joseph Evans added a comment -

          I pulled this into branch-0.23 and branch-2

          Show
          Robert Joseph Evans added a comment - I pulled this into branch-0.23 and branch-2
          Robert Joseph Evans made changes -
          Fix Version/s 0.23.4 [ 12323256 ]
          Fix Version/s 2.0.3-alpha [ 12323272 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #388 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/388/)
          svn merge -c 1382072 FIXES: YARN-57. Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1391096)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391096
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #388 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/388/ ) svn merge -c 1382072 FIXES: YARN-57 . Allow process-tree based resource calculation et al. to be pluggable to support it on multiple platforms. Contributed by Radim Kolar. (Revision 1391096) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1391096 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/LinuxResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorPlugin.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ResourceCalculatorProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestProcfsBasedProcessTree.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Radim Kolar
              Reporter:
              Radim Kolar
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development