Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.0.3-alpha
    • None
    • None

    Description

      Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.

      Attachments

        1. YARN-18-v8.0.patch
          51 kB
          Wenwu Peng
        2. YARN-18-v7.patch
          144 kB
          Junping Du
        3. YARN-18-v7.3.patch
          143 kB
          Junping Du
        4. YARN-18-v7.2.patch
          144 kB
          Junping Du
        5. YARN-18-v7.1.patch
          144 kB
          Junping Du
        6. YARN-18-v6.patch
          99 kB
          Junping Du
        7. YARN-18-v6.4.patch
          99 kB
          Junping Du
        8. YARN-18-v6.3.patch
          99 kB
          Junping Du
        9. YARN-18-v6.2.patch
          99 kB
          Junping Du
        10. YARN-18-v6.1.patch
          99 kB
          Junping Du
        11. YARN-18-v5.patch
          95 kB
          Junping Du
        12. YARN-18-v5.1.patch
          95 kB
          Junping Du
        13. YARN-18-v4.patch
          80 kB
          Junping Du
        14. YARN-18-v4.3.patch
          88 kB
          Junping Du
        15. YARN-18-v4.2.patch
          89 kB
          Junping Du
        16. YARN-18-v4.1.patch
          89 kB
          Junping Du
        17. YARN-18-v3.patch
          51 kB
          Junping Du
        18. YARN-18-v3.2.patch
          47 kB
          Junping Du
        19. YARN-18-v3.1.patch
          51 kB
          Junping Du
        20. YARN-18-v2.patch
          52 kB
          Junping Du
        21. YARN-18.v8.1.patch
          50 kB
          Wenwu Peng
        22. YARN-18.patch
          52 kB
          Junping Du
        23. Pluggable topologies with NodeGroup for YARN.pdf
          186 kB
          Junping Du
        24. MAPREDUCE-4309-v7.patch
          53 kB
          Junping Du
        25. MAPREDUCE-4309-v6.patch
          53 kB
          Junping Du
        26. MAPREDUCE-4309-v5.patch
          53 kB
          Junping Du
        27. MAPREDUCE-4309-v4.patch
          39 kB
          Junping Du
        28. MAPREDUCE-4309-v3.patch
          39 kB
          Junping Du
        29. MAPREDUCE-4309-v2.patch
          29 kB
          Junping Du
        30. MAPREDUCE-4309.patch
          61 kB
          Junping Du
        31. HierachicalTopologyForYARNr1.pdf
          276 kB
          Junping Du
        32. HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
          61 kB
          Junping Du

        Issue Links

        Activity

          johna John Andrunas made changes -
          Issue-Comment-Bulk Replace String [ ~djp ] [ ~junping_du ]
          johna John Andrunas made changes -
          Issue-Comment-Bulk Replace String [ ~djp ] [ ~junping_du ]
          johna John Andrunas made changes -
          Issue-Comment-Bulk Replace String [ ~djp ] [ ~junping_du ]
          junping_du Junping Du added a comment -

          Thanks Wenwu Peng and Abin Shahab for the effort! Last time when I was in a YARN meetup, the feedback from YARN community is that a hierarchical way is more preferable. So I renew the proposal here (see newer one attached), but haven't get chance to prototype a patch. Quite busy recently on other YARN features, so it may take more time for me to have a patch ready. Let me know if you have interest on helping dev/review/test effort for new patches. Thx!

          junping_du Junping Du added a comment - Thanks Wenwu Peng and Abin Shahab for the effort! Last time when I was in a YARN meetup, the feedback from YARN community is that a hierarchical way is more preferable. So I renew the proposal here (see newer one attached), but haven't get chance to prototype a patch. Quite busy recently on other YARN features, so it may take more time for me to have a patch ready. Let me know if you have interest on helping dev/review/test effort for new patches. Thx!
          ashahab Abin Shahab added a comment -

          Ok it merged fine to trunk. I am having issues merging to branch-2

          ashahab Abin Shahab added a comment - Ok it merged fine to trunk. I am having issues merging to branch-2
          ashahab Abin Shahab added a comment - - edited

          Junping Du, and Wenwu Peng, I tried patching this on trunk and also on branch 2, but get some conflicts in both areas, especially in org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo. I'm wondering what I'm doing wrong.
          I am applying the patch like this:

          patch - p1 < YARN-18.patch

          ashahab Abin Shahab added a comment - - edited Junping Du , and Wenwu Peng , I tried patching this on trunk and also on branch 2, but get some conflicts in both areas, especially in org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo. I'm wondering what I'm doing wrong. I am applying the patch like this: patch - p1 < YARN-18 .patch
          gujilangzi Wenwu Peng made changes -
          Attachment YARN-18.v8.1.patch [ 12658617 ]
          gujilangzi Wenwu Peng added a comment -

          YARN-18.v8.1.patch fix the minor comment issue.

          gujilangzi Wenwu Peng added a comment - YARN-18 .v8.1.patch fix the minor comment issue.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12658611/YARN-18-v8.0.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4480//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4480//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658611/YARN-18-v8.0.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4480//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4480//console This message is automatically generated.
          gujilangzi Wenwu Peng made changes -
          Attachment YARN-18-v8.0.patch [ 12658611 ]
          gujilangzi Wenwu Peng added a comment -

          Pass the right file

          gujilangzi Wenwu Peng added a comment - Pass the right file
          gujilangzi Wenwu Peng made changes -
          Attachment YARN-18-v8-1.patch [ 12658610 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12658610/YARN-18-v8-1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4479//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12658610/YARN-18-v8-1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4479//console This message is automatically generated.
          gujilangzi Wenwu Peng made changes -
          Attachment YARN-18-v8-1.patch [ 12658610 ]
          gujilangzi Wenwu Peng added a comment - - edited

          Sync up with latest changes on trunk base on Junping Du's latest patch. Thanks a lot for Junping.

          gujilangzi Wenwu Peng added a comment - - edited Sync up with latest changes on trunk base on Junping Du 's latest patch. Thanks a lot for Junping.
          junping_du Junping Du made changes -
          Link This issue blocks HADOOP-10512 [ HADOOP-10512 ]
          junping_du Junping Du added a comment -

          Hi, according to our discussion in the design lounge at Hadoop Summit 2013, we agree to change previous design of “Pluggable topologies with NodeGroup for YARN” to something new – support a configurable hierarchical topology that makes adding additional locality layers simple. I attached the new version of proposal - HierachicalTopologyForYARNr1.pdf, please help to review and comments. Thx!

          junping_du Junping Du added a comment - Hi, according to our discussion in the design lounge at Hadoop Summit 2013, we agree to change previous design of “Pluggable topologies with NodeGroup for YARN” to something new – support a configurable hierarchical topology that makes adding additional locality layers simple. I attached the new version of proposal - HierachicalTopologyForYARNr1.pdf, please help to review and comments. Thx!
          junping_du Junping Du made changes -
          Description There are several classes in YARN’s container assignment and task scheduling algorithms that relate to data locality which were updated to give preference to running a container on other locality besides node-local and rack-local (like nodegroup-local). This propose to make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class ScheduledRequests was made a package level class to it would be easier to create a subclass, ScheduledRequestsWithNodeGroup. Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.
          junping_du Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          junping_du Junping Du made changes -
          Summary Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology Configurable Hierarchical Topology for YARN
          junping_du Junping Du made changes -
          Attachment HierachicalTopologyForYARNr1.pdf [ 12594353 ]
          junping_du Junping Du added a comment -

          The findbugs warning seems to be unrelated. Luke Lu and Arun Murthy, would you help to review it? Thanks!

          junping_du Junping Du added a comment - The findbugs warning seems to be unrelated. Luke Lu and Arun Murthy , would you help to review it? Thanks!
          junping_du Junping Du made changes -
          Attachment YARN-18-v7.3.patch [ 12587041 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586900/YARN-18-v7.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1172//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1172//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1172//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586900/YARN-18-v7.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1172//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1172//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1172//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v7.2.patch [ 12586900 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586893/YARN-18-v7.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1171//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1171//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586893/YARN-18-v7.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1171//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1171//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v7.1.patch [ 12586893 ]
          junping_du Junping Du added a comment -

          Sync with latest trunk changes.

          junping_du Junping Du added a comment - Sync with latest trunk changes.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586885/YARN-18-v7.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1170//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586885/YARN-18-v7.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1170//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v7.patch [ 12586885 ]
          junping_du Junping Du added a comment -

          In v7 patch, abstract topology related logic within Schedulers (Fifo, Capacity and Fair) into delegation classes to get rid of creating new Schedulers with NodeGroup extension. Also, improve some naming issues like remove "Abstract" in Factory class, etc.

          junping_du Junping Du added a comment - In v7 patch, abstract topology related logic within Schedulers (Fifo, Capacity and Fair) into delegation classes to get rid of creating new Schedulers with NodeGroup extension. Also, improve some naming issues like remove "Abstract" in Factory class, etc.
          junping_du Junping Du added a comment -

          Arun Murthy As our discussion yesterday, let's schedule a call to discuss the specific implementation details of this JIRA and YARN-19. If you think in RM part, xSchedulerWithNodeGroup (x= Capacity, Fifo, Fair) don't make sense, we can get rid of this by abstracting locality related logic to a delegate class which can be pluggable to extend. Does that make sense to you?

          junping_du Junping Du added a comment - Arun Murthy As our discussion yesterday, let's schedule a call to discuss the specific implementation details of this JIRA and YARN-19 . If you think in RM part, xSchedulerWithNodeGroup (x= Capacity, Fifo, Fair) don't make sense, we can get rid of this by abstracting locality related logic to a delegate class which can be pluggable to extend. Does that make sense to you?
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582077/YARN-18-v6.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/882//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/882//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582077/YARN-18-v6.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/882//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/882//console This message is automatically generated.
          junping_du Junping Du added a comment -

          In previous patch (v6.3), forget to include changes (to factory class name) in yarn-default.xml which cause some tests failure in creation of factory. Now fix it (and comments) in v6.4 patch.

          junping_du Junping Du added a comment - In previous patch (v6.3), forget to include changes (to factory class name) in yarn-default.xml which cause some tests failure in creation of factory. Now fix it (and comments) in v6.4 patch.
          junping_du Junping Du made changes -
          Attachment YARN-18-v6.4.patch [ 12582077 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582025/YARN-18-v6.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
          org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager
          org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestRM
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          org.apache.hadoop.yarn.server.resourcemanager.security.TestClientTokens
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/878//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/878//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582025/YARN-18-v6.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestClientTokens org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/878//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/878//console This message is automatically generated.
          jdu Junping Du made changes -
          Attachment YARN-18-v6.3.patch [ 12582025 ]
          jdu Junping Du added a comment -

          Minor changes to address Luke's recently comments.

          jdu Junping Du added a comment - Minor changes to address Luke's recently comments.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581846/YARN-18-v6.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/871//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/871//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581846/YARN-18-v6.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/871//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/871//console This message is automatically generated.
          jdu Junping Du added a comment -

          Arun MurthyYes. All implementation details can be found in this doc. It is dedicated for patch reviewers on YARN-18 and YARN-19. Hope it helps for review and welcome comment.
          Luke LuAgree. Will replace "Elements" with "Aware" in 6.3 patch. Thanks for reviewing carefully.

          jdu Junping Du added a comment - Arun Murthy Yes. All implementation details can be found in this doc. It is dedicated for patch reviewers on YARN-18 and YARN-19 . Hope it helps for review and welcome comment. Luke Lu Agree. Will replace "Elements" with "Aware" in 6.3 patch. Thanks for reviewing carefully.
          vicaya Luke Lu added a comment -

          Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf"

          vicaya Luke Lu added a comment - Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf"
          acmurthy Arun Murthy added a comment -

          Junping, there are lot of changes to digest here.

          Can you please provide a summary/writeup of your patch? This will help me review it. Thanks!

          acmurthy Arun Murthy added a comment - Junping, there are lot of changes to digest here. Can you please provide a summary/writeup of your patch? This will help me review it. Thanks!
          vicaya Luke Lu added a comment -

          The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory?

          This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19.

          vicaya Luke Lu added a comment - The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory? This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19 .
          junping_du Junping Du made changes -
          Attachment YARN-18-v6.2.patch [ 12581846 ]
          junping_du Junping Du added a comment -

          Sync up with latest trunk and do tiny change on RMContainerRequestor so that it doesn't have to be aware of NodeGroup concept.

          junping_du Junping Du added a comment - Sync up with latest trunk and do tiny change on RMContainerRequestor so that it doesn't have to be aware of NodeGroup concept.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581468/YARN-18-v6.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/857//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/857//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581468/YARN-18-v6.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/857//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/857//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v6.1.patch [ 12581468 ]
          junping_du Junping Du added a comment -

          Address minor javadoc issue in v6.1 patch.

          junping_du Junping Du added a comment - Address minor javadoc issue in v6.1 patch.
          gmcdonald Gavin McDonald made changes -
          Link This issue depends upon HADOOP-8469 [ HADOOP-8469 ]
          gmcdonald Gavin McDonald made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          gmcdonald Gavin McDonald made changes -
          Link This issue is depended upon by YARN-19 [ YARN-19 ]
          gmcdonald Gavin McDonald made changes -
          Link This issue blocks YARN-19 [ YARN-19 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581362/YARN-18-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/854//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/854//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581362/YARN-18-v6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/854//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/854//console This message is automatically generated.
          junping_du Junping Du added a comment -

          Thanks Luke for explanation.
          Hey Arun (Arun Murthy), I just attached a doc with implementation details for this patch and YARN-19. Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts?

          junping_du Junping Du added a comment - Thanks Luke for explanation. Hey Arun ( Arun Murthy ), I just attached a doc with implementation details for this patch and YARN-19 . Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts?
          junping_du Junping Du made changes -
          Attachment YARN-18-v6.patch [ 12581362 ]
          junping_du Junping Du added a comment -

          Sync patch to latest trunk which keep consist with doc just attached.

          junping_du Junping Du added a comment - Sync patch to latest trunk which keep consist with doc just attached.
          junping_du Junping Du made changes -
          junping_du Junping Du added a comment -

          Implementation doc for NodeGroup layer support in YARN.

          junping_du Junping Du added a comment - Implementation doc for NodeGroup layer support in YARN.
          vicaya Luke Lu added a comment -

          I see a lot of factories and then no changes to actually support node-group..

          YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19.

          we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology.

          This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology.

          vicaya Luke Lu added a comment - I see a lot of factories and then no changes to actually support node-group.. YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19 . we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology. This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology.
          acmurthy Arun Murthy added a comment - - edited

          Luke Lu / Junping Du - Can you please describe what is happening here? I see a lot of factories and then no changes to actually support node-group...

          What do I need to read-up to understand what is going on here?

          One thought - seems to me, (maybe longer term), we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology... think of complex star-topologies etc.

          Thoughts?

          Another thing to consider is that, for now, it might be better to implement a new NodeGroup Scheduler rather than change all existing Schedulers?

          acmurthy Arun Murthy added a comment - - edited Luke Lu / Junping Du - Can you please describe what is happening here? I see a lot of factories and then no changes to actually support node-group... What do I need to read-up to understand what is going on here? One thought - seems to me, (maybe longer term), we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology... think of complex star-topologies etc. Thoughts? Another thing to consider is that, for now, it might be better to implement a new NodeGroup Scheduler rather than change all existing Schedulers?
          junping_du Junping Du added a comment -

          Thanks Arun and Luke for comments and review.
          Arun Murthy, YARN-18 and YARN-19 are still part of HADOOP-8468. In that umbrella JIRA, I had a proposal to describe how these two changes (P6, P7) works in detail level which may helps you to review this patch. If necessary, I can reflect the current changes (with addressing lots of comments above) and attach again in this JIRA. Thx!

          junping_du Junping Du added a comment - Thanks Arun and Luke for comments and review. Arun Murthy , YARN-18 and YARN-19 are still part of HADOOP-8468 . In that umbrella JIRA, I had a proposal to describe how these two changes (P6, P7) works in detail level which may helps you to review this patch. If necessary, I can reflect the current changes (with addressing lots of comments above) and attach again in this JIRA. Thx!
          vicaya Luke Lu added a comment -

          The original patch were already broken up into YARN-18 and YARN-19. YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests.

          vicaya Luke Lu added a comment - The original patch were already broken up into YARN-18 and YARN-19 . YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests.
          acmurthy Arun Murthy added a comment -

          Sorry, I'm just getting to this. This is a lot to digest.

          Can we consider breaking this down to a couple of smaller patches? Tx.

          acmurthy Arun Murthy added a comment - Sorry, I'm just getting to this. This is a lot to digest. Can we consider breaking this down to a couple of smaller patches? Tx.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575487/YARN-18-v5.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/601//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/601//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575487/YARN-18-v5.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/601//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/601//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v5.1.patch [ 12575487 ]
          junping_du Junping Du added a comment -

          Fix one javadoc issue in v5.1 patch.

          junping_du Junping Du added a comment - Fix one javadoc issue in v5.1 patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575479/YARN-18-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/600//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/600//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575479/YARN-18-v5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/600//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/600//console This message is automatically generated.
          junping_du Junping Du added a comment -

          Incorporate Luke's comments in latest (v5) patch.

          junping_du Junping Du added a comment - Incorporate Luke's comments in latest (v5) patch.
          junping_du Junping Du made changes -
          Attachment YARN-18-v5.patch [ 12575479 ]
          junping_du Junping Du added a comment -

          Thanks Luke for good suggestion. "yarn.app.topology.aware.factory.impl" seems better. Will address your comments in next patch.

          junping_du Junping Du added a comment - Thanks Luke for good suggestion. "yarn.app.topology.aware.factory.impl" seems better. Will address your comments in next patch.
          vicaya Luke Lu added a comment -

          Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps?

          I think we're getting close.

          vicaya Luke Lu added a comment - Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps? I think we're getting close.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574124/YARN-18-v4.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/528//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/528//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574124/YARN-18-v4.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/528//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/528//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v4.3.patch [ 12574124 ]
          junping_du Junping Du added a comment -

          Do minor cleanup on code that should be put in YARN-19.

          junping_du Junping Du added a comment - Do minor cleanup on code that should be put in YARN-19 .
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573476/YARN-18-v4.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/511//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/511//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573476/YARN-18-v4.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/511//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/511//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v4.2.patch [ 12573476 ]
          junping_du Junping Du added a comment -

          Fix the last JavaDoc warning in 4.2 patch.

          junping_du Junping Du added a comment - Fix the last JavaDoc warning in 4.2 patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573471/YARN-18-v4.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/510//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/510//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573471/YARN-18-v4.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/510//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/510//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v4.1.patch [ 12573471 ]
          junping_du Junping Du added a comment -

          Address timeout and JavaDoc issue in v4.1 patch.

          junping_du Junping Du added a comment - Address timeout and JavaDoc issue in v4.1 patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573355/YARN-18-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 9 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/505//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/505//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573355/YARN-18-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 9 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/505//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/505//console This message is automatically generated.
          junping_du Junping Du added a comment -

          Thanks Luke for your review and comments! I addressed almost all your points in V4 patch except for putting ScheduledRequests in topology related factory class for YARN as I think this is the object created within MRAppMaster but not YARN ResourceManager like other objects so in different package. Am I missing something?

          junping_du Junping Du added a comment - Thanks Luke for your review and comments! I addressed almost all your points in V4 patch except for putting ScheduledRequests in topology related factory class for YARN as I think this is the object created within MRAppMaster but not YARN ResourceManager like other objects so in different package. Am I missing something?
          junping_du Junping Du made changes -
          Attachment YARN-18-v4.patch [ 12573355 ]
          vicaya Luke Lu added a comment -

          Preliminary comments:

          1. We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code.
          2. We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects.
          3. The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory.
          4. Pluggable interface/class should be public with the InterfaceAudience.Private annotation.
          vicaya Luke Lu added a comment - Preliminary comments: We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code. We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects. The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory. Pluggable interface/class should be public with the InterfaceAudience.Private annotation.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572023/YARN-18-v3.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/467//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/467//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572023/YARN-18-v3.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/467//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/467//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v3.2.patch [ 12572023 ]
          junping_du Junping Du added a comment -

          Add timeout to all related tests in v3.2 patch.

          junping_du Junping Du added a comment - Add timeout to all related tests in v3.2 patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572014/YARN-18-v3.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/466//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572014/YARN-18-v3.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/466//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/466//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v3.1.patch [ 12572014 ]
          junping_du Junping Du added a comment -

          Add timeout in test at v3.1 patch.

          junping_du Junping Du added a comment - Add timeout in test at v3.1 patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571919/YARN-18-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/459//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/459//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571919/YARN-18-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/459//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/459//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment YARN-18-v3.patch [ 12571919 ]
          junping_du Junping Du added a comment -

          Sync patch with recently changes on YARN.

          junping_du Junping Du added a comment - Sync patch with recently changes on YARN.
          junping_du Junping Du added a comment -

          Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with specific configuration.
          So for this patch, it tries to make locality related interface/class can be extensible with node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class, we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks. Also, this patch add more unit tests. Please let me know if anything here make you confusing. Thanks!

          junping_du Junping Du added a comment - Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with specific configuration. So for this patch, it tries to make locality related interface/class can be extensible with node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class, we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks. Also, this patch add more unit tests. Please let me know if anything here make you confusing. Thanks!
          acmurthy Arun Murthy added a comment -

          Junping, sorry, I've missed this.

          Can you please provide a simple writeup of how you are trying to accomplish this? That will help me review the patch. Thanks.

          acmurthy Arun Murthy added a comment - Junping, sorry, I've missed this. Can you please provide a simple writeup of how you are trying to accomplish this? That will help me review the patch. Thanks.
          junping_du Junping Du added a comment -

          Thanks Tom!

          junping_du Junping Du added a comment - Thanks Tom!
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/187//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/187//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/187//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/187//console This message is automatically generated.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.mapreduce.v2.app.TestAMInfos
          org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
          org.apache.hadoop.mapreduce.v2.app.TestRecovery

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/154//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/154//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.mapreduce.v2.app.TestAMInfos org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler org.apache.hadoop.mapreduce.v2.app.TestRecovery +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/154//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/154//console This message is automatically generated.
          tgraves Thomas Graves added a comment -

          I kicked the pre-commit build.

          tgraves Thomas Graves added a comment - I kicked the pre-commit build.
          junping_du Junping Du added a comment -

          Looks like pre-commit test (https://builds.apache.org/job/PreCommit-YARN-Build/) is not automatically triggered. Is there anything I should do besides set this issue to Patch Available?

          junping_du Junping Du added a comment - Looks like pre-commit test ( https://builds.apache.org/job/PreCommit-YARN-Build/ ) is not automatically triggered. Is there anything I should do besides set this issue to Patch Available?
          junping_du Junping Du made changes -
          Attachment YARN-18-v2.patch [ 12554120 ]
          junping_du Junping Du added a comment -

          Fix minor issues due to rebase work in YARN-18-v2 patch.

          junping_du Junping Du added a comment - Fix minor issues due to rebase work in YARN-18 -v2 patch.
          junping_du Junping Du made changes -
          Affects Version/s 2.0.3-alpha [ 12323272 ]
          Labels features
          Status Open [ 1 ] Patch Available [ 10002 ]
          junping_du Junping Du made changes -
          Attachment YARN-18.patch [ 12554113 ]
          junping_du Junping Du added a comment -

          Rebase to recently code base on YARN.

          junping_du Junping Du added a comment - Rebase to recently code base on YARN.
          gmcdonald Gavin McDonald made changes -
          Workflow jira [ 12720946 ] no-reopen-closed, patch-avail [ 12722153 ]
          acmurthy Arun Murthy made changes -
          Key MAPREDUCE-4309 YARN-18
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Workflow no-reopen-closed, patch-avail [ 12671853 ] jira [ 12720946 ]
          Status Patch Available [ 10002 ] Open [ 1 ]
          acmurthy Arun Murthy made changes -
          Issue Type Bug [ 1 ] New Feature [ 2 ]
          junping_du Junping Du added a comment -

          Bikas, Thanks for your review and help on this. Look forward to incorporate with you again in other JIRA issues.

          junping_du Junping Du added a comment - Bikas, Thanks for your review and help on this. Look forward to incorporate with you again in other JIRA issues.
          bikassaha Bikas Saha added a comment -

          Thanks for incorporating the change for my review comments.
          As far as the scheduler refactoring/factory changes are concerned I will defer to the experts in the scheduler code for final comments.

          bikassaha Bikas Saha added a comment - Thanks for incorporating the change for my review comments. As far as the scheduler refactoring/factory changes are concerned I will defer to the experts in the scheduler code for final comments.
          junping_du Junping Du added a comment -

          Anyone can take a look at this patch. The -1 in findbugs are unrelated and due to FairScheduler. Also, that issue is tracked by MAPREDUCE-4452. Thanks!

          junping_du Junping Du added a comment - Anyone can take a look at this patch. The -1 in findbugs are unrelated and due to FairScheduler. Also, that issue is tracked by MAPREDUCE-4452 . Thanks!
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537557/MAPREDUCE-4309-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537557/MAPREDUCE-4309-v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v7.patch [ 12537557 ]
          junping_du Junping Du added a comment -

          Rebase to latest trunk

          junping_du Junping Du added a comment - Rebase to latest trunk
          junping_du Junping Du added a comment -

          The findbugs issues are unrelated and I already deliver a patch to fix that in MAPREDUCE-4452.

          junping_du Junping Du added a comment - The findbugs issues are unrelated and I already deliver a patch to fix that in MAPREDUCE-4452 .
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537322/MAPREDUCE-4309-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537322/MAPREDUCE-4309-v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v6.patch [ 12537322 ]
          junping_du Junping Du added a comment -

          In newest patch (v6), update to use abstract factory to create some scheduler related objects and rebase to latest trunk.

          junping_du Junping Du added a comment - In newest patch (v6), update to use abstract factory to create some scheduler related objects and rebase to latest trunk.
          szetszwo Tsz-wo Sze added a comment -

          Abstract Factory sounds good.

          For the enum problem, let's use enum for the moment and change it to interfaces later.

          (Sorry that I was not able to response earlier.)

          szetszwo Tsz-wo Sze added a comment - Abstract Factory sounds good. For the enum problem, let's use enum for the moment and change it to interfaces later. (Sorry that I was not able to response earlier.)
          junping_du Junping Du added a comment -

          The FindBugs issue is not related to this patch but related to YARN FairScheduler the checked in recently. I already file a separate JIRA (MAPREDUCE-4452) to fix issues found by FindBugs.

          junping_du Junping Du added a comment - The FindBugs issue is not related to this patch but related to YARN FairScheduler the checked in recently. I already file a separate JIRA ( MAPREDUCE-4452 ) to fix issues found by FindBugs.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536840/MAPREDUCE-4309-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536840/MAPREDUCE-4309-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//console This message is automatically generated.
          junping_du Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v5.patch [ 12536840 ]
          junping_du Junping Du added a comment -

          Merge with recently code changes on YARN.

          junping_du Junping Du added a comment - Merge with recently code changes on YARN.
          junping_du Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          junping_du Junping Du added a comment -

          Looks like MAPREDUCE-4440 is just checked in with a lot of changes on Scheduler. Will cancel the patch until finishing and testing the merge work.

          junping_du Junping Du added a comment - Looks like MAPREDUCE-4440 is just checked in with a lot of changes on Scheduler. Will cancel the patch until finishing and testing the merge work.
          junping_du Junping Du added a comment -

          Yes. That's also what I am thinking. As using configuration based reflection to create each related objects will make configuring work complex and easy to get mistakes for user (also I think most of these objects are belonging to implementation details).
          So, I am thinking of two options based on factory pattern:
          1. Add only one property in configure to mark the topology is with nodegroup, then we have a factory class to construct objects base on configuration property.
          2. Leverage the pattern of "Abstract Factory": use reflection to create factory class according to property in configuration, then use factory to create different objects accordingly.
          I prefer Option 2. What do you think?
          @Nicholas, do you have any comments here?

          junping_du Junping Du added a comment - Yes. That's also what I am thinking. As using configuration based reflection to create each related objects will make configuring work complex and easy to get mistakes for user (also I think most of these objects are belonging to implementation details). So, I am thinking of two options based on factory pattern: 1. Add only one property in configure to mark the topology is with nodegroup, then we have a factory class to construct objects base on configuration property. 2. Leverage the pattern of "Abstract Factory": use reflection to create factory class according to property in configuration, then use factory to create different objects accordingly. I prefer Option 2. What do you think? @Nicholas, do you have any comments here?
          bikassaha Bikas Saha added a comment -

          I am wondering if reflection is the best way to incorporate plugins? Can we replace with a factory pattern or something?

          bikassaha Bikas Saha added a comment - I am wondering if reflection is the best way to incorporate plugins? Can we replace with a factory pattern or something?
          junping_du Junping Du added a comment -

          Bikas, thanks for your comments. Please see my reply:
          1). That's a good point. I have this code before decrementOutstanding() is there. I will fix this in following patch.
          2 and 3). subclass is put in as plug-in patch which is tracked by MAPREDUCE-4310. With seeing that patch as well as the proposal in Umbrella jira (HADOOP-8468), it will help you understand the work we are doing in this jira.

          junping_du Junping Du added a comment - Bikas, thanks for your comments. Please see my reply: 1). That's a good point. I have this code before decrementOutstanding() is there. I will fix this in following patch. 2 and 3). subclass is put in as plug-in patch which is tracked by MAPREDUCE-4310 . With seeing that patch as well as the proposal in Umbrella jira ( HADOOP-8468 ), it will help you understand the work we are doing in this jira.
          bikassaha Bikas Saha added a comment -

          1) AppSchedulingInfo.java
          In allocateNodeGroupLocal(...), why not call decrementOutstanding(requests.get(priority).get(RMNode.ANY)) instead of the following

              // Do not remove ANY
              ResourceRequest offSwitchRequest = requests.get(priority).get(
                  RMNode.ANY);
              offSwitchRequest.setNumContainers(offSwitchRequest.getNumContainers() 
          

          2) Many methods/variables of multilple classes (LeafQueue/FifoScheduler etc) have been made protected but I dont see them subclassed anywhere. Whats the purpose of those changes?

          3) Some objects are not being created via reflection and config. Its not clear what the purpose of those are?

          bikassaha Bikas Saha added a comment - 1) AppSchedulingInfo.java In allocateNodeGroupLocal(...), why not call decrementOutstanding(requests.get(priority).get(RMNode.ANY)) instead of the following // Do not remove ANY ResourceRequest offSwitchRequest = requests.get(priority).get( RMNode.ANY); offSwitchRequest.setNumContainers(offSwitchRequest.getNumContainers() 2) Many methods/variables of multilple classes (LeafQueue/FifoScheduler etc) have been made protected but I dont see them subclassed anywhere. Whats the purpose of those changes? 3) Some objects are not being created via reflection and config. Its not clear what the purpose of those are?
          junping_du Junping Du added a comment -

          Hello Nicholas,
          Thanks for your carefully review. Please see my comments and questions below:
          1. We need to add new element to JobCounter and NodeType at least in pluggin patch (MAPREDUCE-4310). For enum of JobCounter and NodeType, it is hard to extend even in plug-in code as we know. If adding new element to base Enum really hurts, then I am OK with using type-safe enum class (effective java 1st version, http://developer.java.sun.com/developer/Books/effectivejava/Chapter5.pdf) to replace enum if we decide to do it. If we decide to the latter one, it will make patch in-compatible but could be easily extended in future. Which way are we prefer here?
          2. I will put some unused code like: NODEGROUP_LOCAL_MAPS, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP to the plug-in patch. Also, NodeType.NODEGROUP_LOCAL is used twice in this pluggable patch but can be refactored to avoid to appear in this patch:
          1) For one usage, which is in FifoScheduler, we can override getMaxAllocatableContainers() to make NodeType.NODEGROUP_LOCAL go to FifoSchedulerWithNodeGroup.
          2) For the other usage, which is in AppSchedulingInfo, we can use the standard pluggable way to create a subclass of AppSchedulingInfo i.e: AppSchedulingWithNodeGroupInfo and use ReflectionUtils to construct one at round time. However, as we already add some pluggable classes for nodegroup-aware in configuration as following, I am wondering if we are on the right direction to make things pluggable. Does it looks over-complexity for user to configure?

          <code>
          + /** Implementation class of ScheduledRequests.*/
          + public static final String RM_SCHEDULED_REQUESTS_CLASS_KEY =
          + RM_PREFIX + "scheduled.requests.class";
          +
          + /** The boolean value if network topology with nodegroup.*/
          + public static final String NET_TOPOLOGY_WITH_NODEGROUP =
          + "net.topology.with.nodegroup";
          +
          + /** Implementation class of SchedulerNode.*/
          + public static final String RM_SCHEDULER_NODE_CLASS_KEY =
          + RM_PREFIX + "scheduler.node.class";
          +
          + /** LeafQueue implementation class for Capacity Scheduler.*/
          + public static final String RM_CAPACITY_SCHEDULER_LEAFQUEUE_CLASS_KEY =
          + RM_PREFIX + "scheduler.leafqueue.class";
          <code>

          junping_du Junping Du added a comment - Hello Nicholas, Thanks for your carefully review. Please see my comments and questions below: 1. We need to add new element to JobCounter and NodeType at least in pluggin patch ( MAPREDUCE-4310 ). For enum of JobCounter and NodeType, it is hard to extend even in plug-in code as we know. If adding new element to base Enum really hurts, then I am OK with using type-safe enum class (effective java 1st version, http://developer.java.sun.com/developer/Books/effectivejava/Chapter5.pdf ) to replace enum if we decide to do it. If we decide to the latter one, it will make patch in-compatible but could be easily extended in future. Which way are we prefer here? 2. I will put some unused code like: NODEGROUP_LOCAL_MAPS, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP to the plug-in patch. Also, NodeType.NODEGROUP_LOCAL is used twice in this pluggable patch but can be refactored to avoid to appear in this patch: 1) For one usage, which is in FifoScheduler, we can override getMaxAllocatableContainers() to make NodeType.NODEGROUP_LOCAL go to FifoSchedulerWithNodeGroup. 2) For the other usage, which is in AppSchedulingInfo, we can use the standard pluggable way to create a subclass of AppSchedulingInfo i.e: AppSchedulingWithNodeGroupInfo and use ReflectionUtils to construct one at round time. However, as we already add some pluggable classes for nodegroup-aware in configuration as following, I am wondering if we are on the right direction to make things pluggable. Does it looks over-complexity for user to configure? <code> + /** Implementation class of ScheduledRequests.*/ + public static final String RM_SCHEDULED_REQUESTS_CLASS_KEY = + RM_PREFIX + "scheduled.requests.class"; + + /** The boolean value if network topology with nodegroup.*/ + public static final String NET_TOPOLOGY_WITH_NODEGROUP = + "net.topology.with.nodegroup"; + + /** Implementation class of SchedulerNode.*/ + public static final String RM_SCHEDULER_NODE_CLASS_KEY = + RM_PREFIX + "scheduler.node.class"; + + /** LeafQueue implementation class for Capacity Scheduler.*/ + public static final String RM_CAPACITY_SCHEDULER_LEAFQUEUE_CLASS_KEY = + RM_PREFIX + "scheduler.leafqueue.class"; <code>
          szetszwo Tsz-wo Sze added a comment -
          • Since JobCounter is public evolving, I think we cannot add NODEGROUP_LOCAL_MAPS. Also, NODEGROUP_LOCAL_MAPS is not yet used.
          • Similarly, NodeType.NODEGROUP_LOCAL adds node group notation to the original code. Is there anyway to prevent it? I think we may change the enum to an interface but it is a much bigger change.
          • BTW, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP is not used
          szetszwo Tsz-wo Sze added a comment - Since JobCounter is public evolving, I think we cannot add NODEGROUP_LOCAL_MAPS. Also, NODEGROUP_LOCAL_MAPS is not yet used. Similarly, NodeType.NODEGROUP_LOCAL adds node group notation to the original code. Is there anyway to prevent it? I think we may change the enum to an interface but it is a much bigger change. BTW, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP is not used
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535513/MAPREDUCE-4309-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535513/MAPREDUCE-4309-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//console This message is automatically generated.
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v4.patch [ 12535513 ]
          junping_du Junping Du added a comment -

          In v4 patch, fix two minor issues in previous patch:
          1. In getting declared constructor, use class from string rather than getting class from object (the later could be sub class which will be failed to get constructor).
          2. Keep some variables "private" to get rid of inconsistent synchronized issue that found by FindBugs in previous patch.

          junping_du Junping Du added a comment - In v4 patch, fix two minor issues in previous patch: 1. In getting declared constructor, use class from string rather than getting class from object (the later could be sub class which will be failed to get constructor). 2. Keep some variables "private" to get rid of inconsistent synchronized issue that found by FindBugs in previous patch.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535504/MAPREDUCE-4309-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535504/MAPREDUCE-4309-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//console This message is automatically generated.
          junping_du Junping Du added a comment -

          As this patch is mostly a refactoring work which make MapReduce locality based task scheduling pluggable, so I think it is OK to pass previous test. However, I add new test case in v3 patch which check task assignment based on host and rack locality, also correct a minor bug at checkAssignment() in current trunk test code (it is wrong in check host match with assigned.getContainer().getNodeId().toString() but should be assigned.getContainer().getNodeId().getHost()). Fix a v2 bug as after making ScheduledRequest to be a nested class, it seems the constructor reflection will need the outer class as its first parameter.

          junping_du Junping Du added a comment - As this patch is mostly a refactoring work which make MapReduce locality based task scheduling pluggable, so I think it is OK to pass previous test. However, I add new test case in v3 patch which check task assignment based on host and rack locality, also correct a minor bug at checkAssignment() in current trunk test code (it is wrong in check host match with assigned.getContainer().getNodeId().toString() but should be assigned.getContainer().getNodeId().getHost()). Fix a v2 bug as after making ScheduledRequest to be a nested class, it seems the constructor reflection will need the outer class as its first parameter.
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v3.patch [ 12535504 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535376/MAPREDUCE-4309-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2548//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535376/MAPREDUCE-4309-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2548//console This message is automatically generated.
          junping_du Junping Du added a comment -

          Hey Nicholas,
          I just upload the new patch (v2) which should be applied successfully on trunk. In this patch, I keep ScheduledRequests in RMContainerAllocator as nested class. Welcome comments!

          junping_du Junping Du added a comment - Hey Nicholas, I just upload the new patch (v2) which should be applied successfully on trunk. In this patch, I keep ScheduledRequests in RMContainerAllocator as nested class. Welcome comments!
          junping_du Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309-v2.patch [ 12535376 ]
          junping_du Junping Du added a comment -

          Yes. I cancelled the patch and now am doing refactoring work. Like your suggestion (also what we did before on InnerNode in NetworkTopology), I will keep ScheduledRequest stay as inner class. Will update patch and mark "Patch Available" soon.

          junping_du Junping Du added a comment - Yes. I cancelled the patch and now am doing refactoring work. Like your suggestion (also what we did before on InnerNode in NetworkTopology), I will keep ScheduledRequest stay as inner class. Will update patch and mark "Patch Available" soon.
          junping_du Junping Du made changes -
          Attachment MAPREDUCE-4309.patch [ 12535320 ]
          szetszwo Tsz-wo Sze added a comment -

          Quick comments:

          • It does not apply anymore.
            $patch -p0 -i ~/Downloads/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch 
            patching file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
            Hunk #1 FAILED at 18.
            Hunk #2 succeeded at 37 (offset 1 line).
            Hunk #3 FAILED at 45.
            Hunk #4 succeeded at 65 with fuzz 2 (offset 9 lines).
            Hunk #5 succeeded at 80 (offset 9 lines).
            Hunk #6 succeeded at 119 (offset 9 lines).
            Hunk #7 succeeded at 161 (offset 9 lines).
            Hunk #8 succeeded at 548 (offset 10 lines).
            Hunk #9 FAILED at 633.
            3 out of 9 hunks FAILED -- saving rejects to file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java.rej
            ...
            
          • Could we keep ScheduledRequests as an inner class of RMContainerAllocator, i.e. change it to public/protected/package-private static? It will be easier to see the changes.
          szetszwo Tsz-wo Sze added a comment - Quick comments: It does not apply anymore. $patch -p0 -i ~/Downloads/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch patching file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java Hunk #1 FAILED at 18. Hunk #2 succeeded at 37 (offset 1 line). Hunk #3 FAILED at 45. Hunk #4 succeeded at 65 with fuzz 2 (offset 9 lines). Hunk #5 succeeded at 80 (offset 9 lines). Hunk #6 succeeded at 119 (offset 9 lines). Hunk #7 succeeded at 161 (offset 9 lines). Hunk #8 succeeded at 548 (offset 10 lines). Hunk #9 FAILED at 633. 3 out of 9 hunks FAILED -- saving rejects to file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java.rej ... Could we keep ScheduledRequests as an inner class of RMContainerAllocator, i.e. change it to public/protected/package-private static? It will be easier to see the changes.
          junping_du Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          junping_du Junping Du made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          junping_du Junping Du made changes -
          Link This issue blocks MAPREDUCE-4310 [ MAPREDUCE-4310 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//console This message is automatically generated.
          junping_du Junping Du made changes -
          Link This issue is part of HADOOP-8468 [ HADOOP-8468 ]
          junping_du Junping Du made changes -
          Key HADOOP-8474 MAPREDUCE-4309
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 1.0.0 [ 12318244 ]
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          junping_du Junping Du made changes -
          Parent HADOOP-8468 [ 12559114 ]
          Issue Type Sub-task [ 7 ] Bug [ 1 ]
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//console This message is automatically generated.
          junping_du Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          junping_du Junping Du made changes -
          junping_du Junping Du made changes -
          Field Original Value New Value
          Assignee Junping Du [ djp ]
          junping_du Junping Du created issue -

          People

            junping_du Junping Du
            junping_du Junping Du

            Dates

              Created:
              Updated:

              Slack