Hadoop YARN
  1. Hadoop YARN
  2. YARN-18

Configurable Hierarchical Topology for YARN

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.

      1. HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
        61 kB
        Junping Du
      2. HierachicalTopologyForYARNr1.pdf
        276 kB
        Junping Du
      3. MAPREDUCE-4309.patch
        61 kB
        Junping Du
      4. MAPREDUCE-4309-v2.patch
        29 kB
        Junping Du
      5. MAPREDUCE-4309-v3.patch
        39 kB
        Junping Du
      6. MAPREDUCE-4309-v4.patch
        39 kB
        Junping Du
      7. MAPREDUCE-4309-v5.patch
        53 kB
        Junping Du
      8. MAPREDUCE-4309-v6.patch
        53 kB
        Junping Du
      9. MAPREDUCE-4309-v7.patch
        53 kB
        Junping Du
      10. Pluggable topologies with NodeGroup for YARN.pdf
        186 kB
        Junping Du
      11. YARN-18.patch
        52 kB
        Junping Du
      12. YARN-18-v2.patch
        52 kB
        Junping Du
      13. YARN-18-v3.1.patch
        51 kB
        Junping Du
      14. YARN-18-v3.2.patch
        47 kB
        Junping Du
      15. YARN-18-v3.patch
        51 kB
        Junping Du
      16. YARN-18-v4.1.patch
        89 kB
        Junping Du
      17. YARN-18-v4.2.patch
        89 kB
        Junping Du
      18. YARN-18-v4.3.patch
        88 kB
        Junping Du
      19. YARN-18-v4.patch
        80 kB
        Junping Du
      20. YARN-18-v5.1.patch
        95 kB
        Junping Du
      21. YARN-18-v5.patch
        95 kB
        Junping Du
      22. YARN-18-v6.1.patch
        99 kB
        Junping Du
      23. YARN-18-v6.2.patch
        99 kB
        Junping Du
      24. YARN-18-v6.3.patch
        99 kB
        Junping Du
      25. YARN-18-v6.4.patch
        99 kB
        Junping Du
      26. YARN-18-v6.patch
        99 kB
        Junping Du
      27. YARN-18-v7.1.patch
        144 kB
        Junping Du
      28. YARN-18-v7.2.patch
        144 kB
        Junping Du
      29. YARN-18-v7.3.patch
        143 kB
        Junping Du
      30. YARN-18-v7.patch
        144 kB
        Junping Du

        Issue Links

          Activity

          Junping Du made changes -
          Link This issue blocks HADOOP-10512 [ HADOOP-10512 ]
          Hide
          Junping Du added a comment -

          Hi, according to our discussion in the design lounge at Hadoop Summit 2013, we agree to change previous design of “Pluggable topologies with NodeGroup for YARN” to something new – support a configurable hierarchical topology that makes adding additional locality layers simple. I attached the new version of proposal - HierachicalTopologyForYARNr1.pdf, please help to review and comments. Thx!

          Show
          Junping Du added a comment - Hi, according to our discussion in the design lounge at Hadoop Summit 2013, we agree to change previous design of “Pluggable topologies with NodeGroup for YARN” to something new – support a configurable hierarchical topology that makes adding additional locality layers simple. I attached the new version of proposal - HierachicalTopologyForYARNr1.pdf, please help to review and comments. Thx!
          Junping Du made changes -
          Description There are several classes in YARN’s container assignment and task scheduling algorithms that relate to data locality which were updated to give preference to running a container on other locality besides node-local and rack-local (like nodegroup-local). This propose to make these data structure/algorithms pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class ScheduledRequests was made a package level class to it would be easier to create a subclass, ScheduledRequestsWithNodeGroup. Per discussion in the design lounge of Hadoop Summit 2013, we agreed to change the design of “Pluggable topologies with NodeGroup for YARN” to support a configurable hierarchical topology that makes adding additional locality layers simple. Please refer attached doc HierachicalTopologyForYARNr1.pdf for details.
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Summary Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology Configurable Hierarchical Topology for YARN
          Junping Du made changes -
          Attachment HierachicalTopologyForYARNr1.pdf [ 12594353 ]
          Hide
          Junping Du added a comment -

          The findbugs warning seems to be unrelated. Luke Lu and Arun C Murthy, would you help to review it? Thanks!

          Show
          Junping Du added a comment - The findbugs warning seems to be unrelated. Luke Lu and Arun C Murthy , would you help to review it? Thanks!
          Junping Du made changes -
          Attachment YARN-18-v7.3.patch [ 12587041 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586900/YARN-18-v7.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1172//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1172//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1172//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586900/YARN-18-v7.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1172//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1172//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1172//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v7.2.patch [ 12586900 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586893/YARN-18-v7.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1171//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1171//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586893/YARN-18-v7.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1171//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1171//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v7.1.patch [ 12586893 ]
          Hide
          Junping Du added a comment -

          Sync with latest trunk changes.

          Show
          Junping Du added a comment - Sync with latest trunk changes.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12586885/YARN-18-v7.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1170//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586885/YARN-18-v7.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1170//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v7.patch [ 12586885 ]
          Hide
          Junping Du added a comment -

          In v7 patch, abstract topology related logic within Schedulers (Fifo, Capacity and Fair) into delegation classes to get rid of creating new Schedulers with NodeGroup extension. Also, improve some naming issues like remove "Abstract" in Factory class, etc.

          Show
          Junping Du added a comment - In v7 patch, abstract topology related logic within Schedulers (Fifo, Capacity and Fair) into delegation classes to get rid of creating new Schedulers with NodeGroup extension. Also, improve some naming issues like remove "Abstract" in Factory class, etc.
          Hide
          Junping Du added a comment -

          Arun C Murthy As our discussion yesterday, let's schedule a call to discuss the specific implementation details of this JIRA and YARN-19. If you think in RM part, xSchedulerWithNodeGroup (x= Capacity, Fifo, Fair) don't make sense, we can get rid of this by abstracting locality related logic to a delegate class which can be pluggable to extend. Does that make sense to you?

          Show
          Junping Du added a comment - Arun C Murthy As our discussion yesterday, let's schedule a call to discuss the specific implementation details of this JIRA and YARN-19 . If you think in RM part, xSchedulerWithNodeGroup (x= Capacity, Fifo, Fair) don't make sense, we can get rid of this by abstracting locality related logic to a delegate class which can be pluggable to extend. Does that make sense to you?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582077/YARN-18-v6.4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/882//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/882//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582077/YARN-18-v6.4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/882//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/882//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          In previous patch (v6.3), forget to include changes (to factory class name) in yarn-default.xml which cause some tests failure in creation of factory. Now fix it (and comments) in v6.4 patch.

          Show
          Junping Du added a comment - In previous patch (v6.3), forget to include changes (to factory class name) in yarn-default.xml which cause some tests failure in creation of factory. Now fix it (and comments) in v6.4 patch.
          Junping Du made changes -
          Attachment YARN-18-v6.4.patch [ 12582077 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582025/YARN-18-v6.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
          org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager
          org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestRM
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          org.apache.hadoop.yarn.server.resourcemanager.security.TestClientTokens
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue
          org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
          org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
          org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
          org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/878//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/878//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582025/YARN-18-v6.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestClientTokens org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/878//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/878//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v6.3.patch [ 12582025 ]
          Hide
          Junping Du added a comment -

          Minor changes to address Luke's recently comments.

          Show
          Junping Du added a comment - Minor changes to address Luke's recently comments.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581846/YARN-18-v6.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/871//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/871//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581846/YARN-18-v6.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/871//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/871//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Arun C MurthyYes. All implementation details can be found in this doc. It is dedicated for patch reviewers on YARN-18 and YARN-19. Hope it helps for review and welcome comment.
          Luke LuAgree. Will replace "Elements" with "Aware" in 6.3 patch. Thanks for reviewing carefully.

          Show
          Junping Du added a comment - Arun C Murthy Yes. All implementation details can be found in this doc. It is dedicated for patch reviewers on YARN-18 and YARN-19 . Hope it helps for review and welcome comment. Luke Lu Agree. Will replace "Elements" with "Aware" in 6.3 patch. Thanks for reviewing carefully.
          Hide
          Luke Lu added a comment -

          Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf"

          Show
          Luke Lu added a comment - Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf"
          Hide
          Arun C Murthy added a comment -

          Junping, there are lot of changes to digest here.

          Can you please provide a summary/writeup of your patch? This will help me review it. Thanks!

          Show
          Arun C Murthy added a comment - Junping, there are lot of changes to digest here. Can you please provide a summary/writeup of your patch? This will help me review it. Thanks!
          Hide
          Luke Lu added a comment -

          The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory?

          This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19.

          Show
          Luke Lu added a comment - The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory? This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19 .
          Junping Du made changes -
          Attachment YARN-18-v6.2.patch [ 12581846 ]
          Hide
          Junping Du added a comment -

          Sync up with latest trunk and do tiny change on RMContainerRequestor so that it doesn't have to be aware of NodeGroup concept.

          Show
          Junping Du added a comment - Sync up with latest trunk and do tiny change on RMContainerRequestor so that it doesn't have to be aware of NodeGroup concept.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581468/YARN-18-v6.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/857//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/857//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581468/YARN-18-v6.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/857//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/857//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v6.1.patch [ 12581468 ]
          Hide
          Junping Du added a comment -

          Address minor javadoc issue in v6.1 patch.

          Show
          Junping Du added a comment - Address minor javadoc issue in v6.1 patch.
          Gavin made changes -
          Link This issue depends upon HADOOP-8469 [ HADOOP-8469 ]
          Gavin made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          Gavin made changes -
          Link This issue is depended upon by YARN-19 [ YARN-19 ]
          Gavin made changes -
          Link This issue blocks YARN-19 [ YARN-19 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12581362/YARN-18-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/854//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/854//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581362/YARN-18-v6.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/854//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/854//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Luke for explanation.
          Hey Arun (Arun C Murthy), I just attached a doc with implementation details for this patch and YARN-19. Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts?

          Show
          Junping Du added a comment - Thanks Luke for explanation. Hey Arun ( Arun C Murthy ), I just attached a doc with implementation details for this patch and YARN-19 . Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts?
          Junping Du made changes -
          Attachment YARN-18-v6.patch [ 12581362 ]
          Hide
          Junping Du added a comment -

          Sync patch to latest trunk which keep consist with doc just attached.

          Show
          Junping Du added a comment - Sync patch to latest trunk which keep consist with doc just attached.
          Junping Du made changes -
          Hide
          Junping Du added a comment -

          Implementation doc for NodeGroup layer support in YARN.

          Show
          Junping Du added a comment - Implementation doc for NodeGroup layer support in YARN.
          Hide
          Luke Lu added a comment -

          I see a lot of factories and then no changes to actually support node-group..

          YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19.

          we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology.

          This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology.

          Show
          Luke Lu added a comment - I see a lot of factories and then no changes to actually support node-group.. YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19 . we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology. This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology.
          Hide
          Arun C Murthy added a comment -

          Luke Lu / Junping Du - Can you please describe what is happening here? I see a lot of factories and then no changes to actually support node-group...

          What do I need to read-up to understand what is going on here?

          One thought - seems to me, (maybe longer term), we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology... think of complex star-topologies etc.

          Thoughts?

          Another thing to consider is that, for now, it might be better to implement a new NodeGroup Scheduler rather than change all existing Schedulers?

          Show
          Arun C Murthy added a comment - Luke Lu / Junping Du - Can you please describe what is happening here? I see a lot of factories and then no changes to actually support node-group... What do I need to read-up to understand what is going on here? One thought - seems to me, (maybe longer term), we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology... think of complex star-topologies etc. Thoughts? Another thing to consider is that, for now, it might be better to implement a new NodeGroup Scheduler rather than change all existing Schedulers?
          Hide
          Junping Du added a comment -

          Thanks Arun and Luke for comments and review.
          Arun C Murthy, YARN-18 and YARN-19 are still part of HADOOP-8468. In that umbrella JIRA, I had a proposal to describe how these two changes (P6, P7) works in detail level which may helps you to review this patch. If necessary, I can reflect the current changes (with addressing lots of comments above) and attach again in this JIRA. Thx!

          Show
          Junping Du added a comment - Thanks Arun and Luke for comments and review. Arun C Murthy , YARN-18 and YARN-19 are still part of HADOOP-8468 . In that umbrella JIRA, I had a proposal to describe how these two changes (P6, P7) works in detail level which may helps you to review this patch. If necessary, I can reflect the current changes (with addressing lots of comments above) and attach again in this JIRA. Thx!
          Hide
          Luke Lu added a comment -

          The original patch were already broken up into YARN-18 and YARN-19. YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests.

          Show
          Luke Lu added a comment - The original patch were already broken up into YARN-18 and YARN-19 . YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests.
          Hide
          Arun C Murthy added a comment -

          Sorry, I'm just getting to this. This is a lot to digest.

          Can we consider breaking this down to a couple of smaller patches? Tx.

          Show
          Arun C Murthy added a comment - Sorry, I'm just getting to this. This is a lot to digest. Can we consider breaking this down to a couple of smaller patches? Tx.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575487/YARN-18-v5.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/601//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/601//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575487/YARN-18-v5.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/601//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/601//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v5.1.patch [ 12575487 ]
          Hide
          Junping Du added a comment -

          Fix one javadoc issue in v5.1 patch.

          Show
          Junping Du added a comment - Fix one javadoc issue in v5.1 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12575479/YARN-18-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/600//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/600//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575479/YARN-18-v5.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/600//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/600//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Incorporate Luke's comments in latest (v5) patch.

          Show
          Junping Du added a comment - Incorporate Luke's comments in latest (v5) patch.
          Junping Du made changes -
          Attachment YARN-18-v5.patch [ 12575479 ]
          Hide
          Junping Du added a comment -

          Thanks Luke for good suggestion. "yarn.app.topology.aware.factory.impl" seems better. Will address your comments in next patch.

          Show
          Junping Du added a comment - Thanks Luke for good suggestion. "yarn.app.topology.aware.factory.impl" seems better. Will address your comments in next patch.
          Hide
          Luke Lu added a comment -

          Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps?

          I think we're getting close.

          Show
          Luke Lu added a comment - Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps? I think we're getting close.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12574124/YARN-18-v4.3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/528//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/528//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574124/YARN-18-v4.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/528//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/528//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v4.3.patch [ 12574124 ]
          Hide
          Junping Du added a comment -

          Do minor cleanup on code that should be put in YARN-19.

          Show
          Junping Du added a comment - Do minor cleanup on code that should be put in YARN-19 .
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573476/YARN-18-v4.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/511//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/511//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573476/YARN-18-v4.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/511//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/511//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v4.2.patch [ 12573476 ]
          Hide
          Junping Du added a comment -

          Fix the last JavaDoc warning in 4.2 patch.

          Show
          Junping Du added a comment - Fix the last JavaDoc warning in 4.2 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573471/YARN-18-v4.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/510//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/510//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573471/YARN-18-v4.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/510//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/510//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v4.1.patch [ 12573471 ]
          Hide
          Junping Du added a comment -

          Address timeout and JavaDoc issue in v4.1 patch.

          Show
          Junping Du added a comment - Address timeout and JavaDoc issue in v4.1 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12573355/YARN-18-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 9 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/505//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/505//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573355/YARN-18-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 9 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/505//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/505//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Thanks Luke for your review and comments! I addressed almost all your points in V4 patch except for putting ScheduledRequests in topology related factory class for YARN as I think this is the object created within MRAppMaster but not YARN ResourceManager like other objects so in different package. Am I missing something?

          Show
          Junping Du added a comment - Thanks Luke for your review and comments! I addressed almost all your points in V4 patch except for putting ScheduledRequests in topology related factory class for YARN as I think this is the object created within MRAppMaster but not YARN ResourceManager like other objects so in different package. Am I missing something?
          Junping Du made changes -
          Attachment YARN-18-v4.patch [ 12573355 ]
          Hide
          Luke Lu added a comment -

          Preliminary comments:

          1. We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code.
          2. We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects.
          3. The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory.
          4. Pluggable interface/class should be public with the InterfaceAudience.Private annotation.
          Show
          Luke Lu added a comment - Preliminary comments: We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code. We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects. The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory. Pluggable interface/class should be public with the InterfaceAudience.Private annotation.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572023/YARN-18-v3.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/467//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/467//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572023/YARN-18-v3.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/467//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/467//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v3.2.patch [ 12572023 ]
          Hide
          Junping Du added a comment -

          Add timeout to all related tests in v3.2 patch.

          Show
          Junping Du added a comment - Add timeout to all related tests in v3.2 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572014/YARN-18-v3.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/466//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572014/YARN-18-v3.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/466//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/466//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v3.1.patch [ 12572014 ]
          Hide
          Junping Du added a comment -

          Add timeout in test at v3.1 patch.

          Show
          Junping Du added a comment - Add timeout in test at v3.1 patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12571919/YARN-18-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 one of tests included doesn't have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/459//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/459//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571919/YARN-18-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/459//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/459//console This message is automatically generated.
          Junping Du made changes -
          Attachment YARN-18-v3.patch [ 12571919 ]
          Hide
          Junping Du added a comment -

          Sync patch with recently changes on YARN.

          Show
          Junping Du added a comment - Sync patch with recently changes on YARN.
          Hide
          Junping Du added a comment -

          Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with specific configuration.
          So for this patch, it tries to make locality related interface/class can be extensible with node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class, we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks. Also, this patch add more unit tests. Please let me know if anything here make you confusing. Thanks!

          Show
          Junping Du added a comment - Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with specific configuration. So for this patch, it tries to make locality related interface/class can be extensible with node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class, we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks. Also, this patch add more unit tests. Please let me know if anything here make you confusing. Thanks!
          Hide
          Arun C Murthy added a comment -

          Junping, sorry, I've missed this.

          Can you please provide a simple writeup of how you are trying to accomplish this? That will help me review the patch. Thanks.

          Show
          Arun C Murthy added a comment - Junping, sorry, I've missed this. Can you please provide a simple writeup of how you are trying to accomplish this? That will help me review the patch. Thanks.
          Hide
          Junping Du added a comment -

          Thanks Tom!

          Show
          Junping Du added a comment - Thanks Tom!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/187//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/187//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/187//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/187//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.mapreduce.v2.app.TestAMInfos
          org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
          org.apache.hadoop.mapreduce.v2.app.TestRecovery

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/154//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/154//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.mapreduce.v2.app.TestAMInfos org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler org.apache.hadoop.mapreduce.v2.app.TestRecovery +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/154//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/154//console This message is automatically generated.
          Hide
          Thomas Graves added a comment -

          I kicked the pre-commit build.

          Show
          Thomas Graves added a comment - I kicked the pre-commit build.
          Hide
          Junping Du added a comment -

          Looks like pre-commit test (https://builds.apache.org/job/PreCommit-YARN-Build/) is not automatically triggered. Is there anything I should do besides set this issue to Patch Available?

          Show
          Junping Du added a comment - Looks like pre-commit test ( https://builds.apache.org/job/PreCommit-YARN-Build/ ) is not automatically triggered. Is there anything I should do besides set this issue to Patch Available?
          Junping Du made changes -
          Attachment YARN-18-v2.patch [ 12554120 ]
          Hide
          Junping Du added a comment -

          Fix minor issues due to rebase work in YARN-18-v2 patch.

          Show
          Junping Du added a comment - Fix minor issues due to rebase work in YARN-18 -v2 patch.
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 2.0.3-alpha [ 12323272 ]
          Labels features
          Junping Du made changes -
          Attachment YARN-18.patch [ 12554113 ]
          Hide
          Junping Du added a comment -

          Rebase to recently code base on YARN.

          Show
          Junping Du added a comment - Rebase to recently code base on YARN.
          Gavin made changes -
          Workflow jira [ 12720946 ] no-reopen-closed, patch-avail [ 12722153 ]
          Arun C Murthy made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Key MAPREDUCE-4309 YARN-18
          Workflow no-reopen-closed, patch-avail [ 12671853 ] jira [ 12720946 ]
          Status Patch Available [ 10002 ] Open [ 1 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Arun C Murthy made changes -
          Issue Type Bug [ 1 ] New Feature [ 2 ]
          Hide
          Junping Du added a comment -

          Bikas, Thanks for your review and help on this. Look forward to incorporate with you again in other JIRA issues.

          Show
          Junping Du added a comment - Bikas, Thanks for your review and help on this. Look forward to incorporate with you again in other JIRA issues.
          Hide
          Bikas Saha added a comment -

          Thanks for incorporating the change for my review comments.
          As far as the scheduler refactoring/factory changes are concerned I will defer to the experts in the scheduler code for final comments.

          Show
          Bikas Saha added a comment - Thanks for incorporating the change for my review comments. As far as the scheduler refactoring/factory changes are concerned I will defer to the experts in the scheduler code for final comments.
          Hide
          Junping Du added a comment -

          Anyone can take a look at this patch. The -1 in findbugs are unrelated and due to FairScheduler. Also, that issue is tracked by MAPREDUCE-4452. Thanks!

          Show
          Junping Du added a comment - Anyone can take a look at this patch. The -1 in findbugs are unrelated and due to FairScheduler. Also, that issue is tracked by MAPREDUCE-4452 . Thanks!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537557/MAPREDUCE-4309-v7.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537557/MAPREDUCE-4309-v7.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2648//console This message is automatically generated.
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v7.patch [ 12537557 ]
          Hide
          Junping Du added a comment -

          Rebase to latest trunk

          Show
          Junping Du added a comment - Rebase to latest trunk
          Hide
          Junping Du added a comment -

          The findbugs issues are unrelated and I already deliver a patch to fix that in MAPREDUCE-4452.

          Show
          Junping Du added a comment - The findbugs issues are unrelated and I already deliver a patch to fix that in MAPREDUCE-4452 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537322/MAPREDUCE-4309-v6.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537322/MAPREDUCE-4309-v6.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2634//console This message is automatically generated.
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v6.patch [ 12537322 ]
          Hide
          Junping Du added a comment -

          In newest patch (v6), update to use abstract factory to create some scheduler related objects and rebase to latest trunk.

          Show
          Junping Du added a comment - In newest patch (v6), update to use abstract factory to create some scheduler related objects and rebase to latest trunk.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Abstract Factory sounds good.

          For the enum problem, let's use enum for the moment and change it to interfaces later.

          (Sorry that I was not able to response earlier.)

          Show
          Tsz Wo Nicholas Sze added a comment - Abstract Factory sounds good. For the enum problem, let's use enum for the moment and change it to interfaces later. (Sorry that I was not able to response earlier.)
          Hide
          Junping Du added a comment -

          The FindBugs issue is not related to this patch but related to YARN FairScheduler the checked in recently. I already file a separate JIRA (MAPREDUCE-4452) to fix issues found by FindBugs.

          Show
          Junping Du added a comment - The FindBugs issue is not related to this patch but related to YARN FairScheduler the checked in recently. I already file a separate JIRA ( MAPREDUCE-4452 ) to fix issues found by FindBugs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536840/MAPREDUCE-4309-v5.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536840/MAPREDUCE-4309-v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2612//console This message is automatically generated.
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v5.patch [ 12536840 ]
          Hide
          Junping Du added a comment -

          Merge with recently code changes on YARN.

          Show
          Junping Du added a comment - Merge with recently code changes on YARN.
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Junping Du added a comment -

          Looks like MAPREDUCE-4440 is just checked in with a lot of changes on Scheduler. Will cancel the patch until finishing and testing the merge work.

          Show
          Junping Du added a comment - Looks like MAPREDUCE-4440 is just checked in with a lot of changes on Scheduler. Will cancel the patch until finishing and testing the merge work.
          Hide
          Junping Du added a comment -

          Yes. That's also what I am thinking. As using configuration based reflection to create each related objects will make configuring work complex and easy to get mistakes for user (also I think most of these objects are belonging to implementation details).
          So, I am thinking of two options based on factory pattern:
          1. Add only one property in configure to mark the topology is with nodegroup, then we have a factory class to construct objects base on configuration property.
          2. Leverage the pattern of "Abstract Factory": use reflection to create factory class according to property in configuration, then use factory to create different objects accordingly.
          I prefer Option 2. What do you think?
          @Nicholas, do you have any comments here?

          Show
          Junping Du added a comment - Yes. That's also what I am thinking. As using configuration based reflection to create each related objects will make configuring work complex and easy to get mistakes for user (also I think most of these objects are belonging to implementation details). So, I am thinking of two options based on factory pattern: 1. Add only one property in configure to mark the topology is with nodegroup, then we have a factory class to construct objects base on configuration property. 2. Leverage the pattern of "Abstract Factory": use reflection to create factory class according to property in configuration, then use factory to create different objects accordingly. I prefer Option 2. What do you think? @Nicholas, do you have any comments here?
          Hide
          Bikas Saha added a comment -

          I am wondering if reflection is the best way to incorporate plugins? Can we replace with a factory pattern or something?

          Show
          Bikas Saha added a comment - I am wondering if reflection is the best way to incorporate plugins? Can we replace with a factory pattern or something?
          Hide
          Junping Du added a comment -

          Bikas, thanks for your comments. Please see my reply:
          1). That's a good point. I have this code before decrementOutstanding() is there. I will fix this in following patch.
          2 and 3). subclass is put in as plug-in patch which is tracked by MAPREDUCE-4310. With seeing that patch as well as the proposal in Umbrella jira (HADOOP-8468), it will help you understand the work we are doing in this jira.

          Show
          Junping Du added a comment - Bikas, thanks for your comments. Please see my reply: 1). That's a good point. I have this code before decrementOutstanding() is there. I will fix this in following patch. 2 and 3). subclass is put in as plug-in patch which is tracked by MAPREDUCE-4310 . With seeing that patch as well as the proposal in Umbrella jira ( HADOOP-8468 ), it will help you understand the work we are doing in this jira.
          Hide
          Bikas Saha added a comment -

          1) AppSchedulingInfo.java
          In allocateNodeGroupLocal(...), why not call decrementOutstanding(requests.get(priority).get(RMNode.ANY)) instead of the following

              // Do not remove ANY
              ResourceRequest offSwitchRequest = requests.get(priority).get(
                  RMNode.ANY);
              offSwitchRequest.setNumContainers(offSwitchRequest.getNumContainers() 
          

          2) Many methods/variables of multilple classes (LeafQueue/FifoScheduler etc) have been made protected but I dont see them subclassed anywhere. Whats the purpose of those changes?

          3) Some objects are not being created via reflection and config. Its not clear what the purpose of those are?

          Show
          Bikas Saha added a comment - 1) AppSchedulingInfo.java In allocateNodeGroupLocal(...), why not call decrementOutstanding(requests.get(priority).get(RMNode.ANY)) instead of the following // Do not remove ANY ResourceRequest offSwitchRequest = requests.get(priority).get( RMNode.ANY); offSwitchRequest.setNumContainers(offSwitchRequest.getNumContainers() 2) Many methods/variables of multilple classes (LeafQueue/FifoScheduler etc) have been made protected but I dont see them subclassed anywhere. Whats the purpose of those changes? 3) Some objects are not being created via reflection and config. Its not clear what the purpose of those are?
          Hide
          Junping Du added a comment -

          Hello Nicholas,
          Thanks for your carefully review. Please see my comments and questions below:
          1. We need to add new element to JobCounter and NodeType at least in pluggin patch (MAPREDUCE-4310). For enum of JobCounter and NodeType, it is hard to extend even in plug-in code as we know. If adding new element to base Enum really hurts, then I am OK with using type-safe enum class (effective java 1st version, http://developer.java.sun.com/developer/Books/effectivejava/Chapter5.pdf) to replace enum if we decide to do it. If we decide to the latter one, it will make patch in-compatible but could be easily extended in future. Which way are we prefer here?
          2. I will put some unused code like: NODEGROUP_LOCAL_MAPS, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP to the plug-in patch. Also, NodeType.NODEGROUP_LOCAL is used twice in this pluggable patch but can be refactored to avoid to appear in this patch:
          1) For one usage, which is in FifoScheduler, we can override getMaxAllocatableContainers() to make NodeType.NODEGROUP_LOCAL go to FifoSchedulerWithNodeGroup.
          2) For the other usage, which is in AppSchedulingInfo, we can use the standard pluggable way to create a subclass of AppSchedulingInfo i.e: AppSchedulingWithNodeGroupInfo and use ReflectionUtils to construct one at round time. However, as we already add some pluggable classes for nodegroup-aware in configuration as following, I am wondering if we are on the right direction to make things pluggable. Does it looks over-complexity for user to configure?

          <code>
          + /** Implementation class of ScheduledRequests.*/
          + public static final String RM_SCHEDULED_REQUESTS_CLASS_KEY =
          + RM_PREFIX + "scheduled.requests.class";
          +
          + /** The boolean value if network topology with nodegroup.*/
          + public static final String NET_TOPOLOGY_WITH_NODEGROUP =
          + "net.topology.with.nodegroup";
          +
          + /** Implementation class of SchedulerNode.*/
          + public static final String RM_SCHEDULER_NODE_CLASS_KEY =
          + RM_PREFIX + "scheduler.node.class";
          +
          + /** LeafQueue implementation class for Capacity Scheduler.*/
          + public static final String RM_CAPACITY_SCHEDULER_LEAFQUEUE_CLASS_KEY =
          + RM_PREFIX + "scheduler.leafqueue.class";
          <code>

          Show
          Junping Du added a comment - Hello Nicholas, Thanks for your carefully review. Please see my comments and questions below: 1. We need to add new element to JobCounter and NodeType at least in pluggin patch ( MAPREDUCE-4310 ). For enum of JobCounter and NodeType, it is hard to extend even in plug-in code as we know. If adding new element to base Enum really hurts, then I am OK with using type-safe enum class (effective java 1st version, http://developer.java.sun.com/developer/Books/effectivejava/Chapter5.pdf ) to replace enum if we decide to do it. If we decide to the latter one, it will make patch in-compatible but could be easily extended in future. Which way are we prefer here? 2. I will put some unused code like: NODEGROUP_LOCAL_MAPS, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP to the plug-in patch. Also, NodeType.NODEGROUP_LOCAL is used twice in this pluggable patch but can be refactored to avoid to appear in this patch: 1) For one usage, which is in FifoScheduler, we can override getMaxAllocatableContainers() to make NodeType.NODEGROUP_LOCAL go to FifoSchedulerWithNodeGroup. 2) For the other usage, which is in AppSchedulingInfo, we can use the standard pluggable way to create a subclass of AppSchedulingInfo i.e: AppSchedulingWithNodeGroupInfo and use ReflectionUtils to construct one at round time. However, as we already add some pluggable classes for nodegroup-aware in configuration as following, I am wondering if we are on the right direction to make things pluggable. Does it looks over-complexity for user to configure? <code> + /** Implementation class of ScheduledRequests.*/ + public static final String RM_SCHEDULED_REQUESTS_CLASS_KEY = + RM_PREFIX + "scheduled.requests.class"; + + /** The boolean value if network topology with nodegroup.*/ + public static final String NET_TOPOLOGY_WITH_NODEGROUP = + "net.topology.with.nodegroup"; + + /** Implementation class of SchedulerNode.*/ + public static final String RM_SCHEDULER_NODE_CLASS_KEY = + RM_PREFIX + "scheduler.node.class"; + + /** LeafQueue implementation class for Capacity Scheduler.*/ + public static final String RM_CAPACITY_SCHEDULER_LEAFQUEUE_CLASS_KEY = + RM_PREFIX + "scheduler.leafqueue.class"; <code>
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • Since JobCounter is public evolving, I think we cannot add NODEGROUP_LOCAL_MAPS. Also, NODEGROUP_LOCAL_MAPS is not yet used.
          • Similarly, NodeType.NODEGROUP_LOCAL adds node group notation to the original code. Is there anyway to prevent it? I think we may change the enum to an interface but it is a much bigger change.
          • BTW, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP is not used
          Show
          Tsz Wo Nicholas Sze added a comment - Since JobCounter is public evolving, I think we cannot add NODEGROUP_LOCAL_MAPS. Also, NODEGROUP_LOCAL_MAPS is not yet used. Similarly, NodeType.NODEGROUP_LOCAL adds node group notation to the original code. Is there anyway to prevent it? I think we may change the enum to an interface but it is a much bigger change. BTW, YarnConfiguration.NET_TOPOLOGY_WITH_NODEGROUP is not used
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535513/MAPREDUCE-4309-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535513/MAPREDUCE-4309-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2554//console This message is automatically generated.
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v4.patch [ 12535513 ]
          Hide
          Junping Du added a comment -

          In v4 patch, fix two minor issues in previous patch:
          1. In getting declared constructor, use class from string rather than getting class from object (the later could be sub class which will be failed to get constructor).
          2. Keep some variables "private" to get rid of inconsistent synchronized issue that found by FindBugs in previous patch.

          Show
          Junping Du added a comment - In v4 patch, fix two minor issues in previous patch: 1. In getting declared constructor, use class from string rather than getting class from object (the later could be sub class which will be failed to get constructor). 2. Keep some variables "private" to get rid of inconsistent synchronized issue that found by FindBugs in previous patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535504/MAPREDUCE-4309-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535504/MAPREDUCE-4309-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2553//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          As this patch is mostly a refactoring work which make MapReduce locality based task scheduling pluggable, so I think it is OK to pass previous test. However, I add new test case in v3 patch which check task assignment based on host and rack locality, also correct a minor bug at checkAssignment() in current trunk test code (it is wrong in check host match with assigned.getContainer().getNodeId().toString() but should be assigned.getContainer().getNodeId().getHost()). Fix a v2 bug as after making ScheduledRequest to be a nested class, it seems the constructor reflection will need the outer class as its first parameter.

          Show
          Junping Du added a comment - As this patch is mostly a refactoring work which make MapReduce locality based task scheduling pluggable, so I think it is OK to pass previous test. However, I add new test case in v3 patch which check task assignment based on host and rack locality, also correct a minor bug at checkAssignment() in current trunk test code (it is wrong in check host match with assigned.getContainer().getNodeId().toString() but should be assigned.getContainer().getNodeId().getHost()). Fix a v2 bug as after making ScheduledRequest to be a nested class, it seems the constructor reflection will need the outer class as its first parameter.
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v3.patch [ 12535504 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535376/MAPREDUCE-4309-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2548//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535376/MAPREDUCE-4309-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2548//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          Hey Nicholas,
          I just upload the new patch (v2) which should be applied successfully on trunk. In this patch, I keep ScheduledRequests in RMContainerAllocator as nested class. Welcome comments!

          Show
          Junping Du added a comment - Hey Nicholas, I just upload the new patch (v2) which should be applied successfully on trunk. In this patch, I keep ScheduledRequests in RMContainerAllocator as nested class. Welcome comments!
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Attachment MAPREDUCE-4309-v2.patch [ 12535376 ]
          Hide
          Junping Du added a comment -

          Yes. I cancelled the patch and now am doing refactoring work. Like your suggestion (also what we did before on InnerNode in NetworkTopology), I will keep ScheduledRequest stay as inner class. Will update patch and mark "Patch Available" soon.

          Show
          Junping Du added a comment - Yes. I cancelled the patch and now am doing refactoring work. Like your suggestion (also what we did before on InnerNode in NetworkTopology), I will keep ScheduledRequest stay as inner class. Will update patch and mark "Patch Available" soon.
          Junping Du made changes -
          Attachment MAPREDUCE-4309.patch [ 12535320 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Quick comments:

          • It does not apply anymore.
            $patch -p0 -i ~/Downloads/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch 
            patching file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
            Hunk #1 FAILED at 18.
            Hunk #2 succeeded at 37 (offset 1 line).
            Hunk #3 FAILED at 45.
            Hunk #4 succeeded at 65 with fuzz 2 (offset 9 lines).
            Hunk #5 succeeded at 80 (offset 9 lines).
            Hunk #6 succeeded at 119 (offset 9 lines).
            Hunk #7 succeeded at 161 (offset 9 lines).
            Hunk #8 succeeded at 548 (offset 10 lines).
            Hunk #9 FAILED at 633.
            3 out of 9 hunks FAILED -- saving rejects to file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java.rej
            ...
            
          • Could we keep ScheduledRequests as an inner class of RMContainerAllocator, i.e. change it to public/protected/package-private static? It will be easier to see the changes.
          Show
          Tsz Wo Nicholas Sze added a comment - Quick comments: It does not apply anymore. $patch -p0 -i ~/Downloads/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch patching file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java Hunk #1 FAILED at 18. Hunk #2 succeeded at 37 (offset 1 line). Hunk #3 FAILED at 45. Hunk #4 succeeded at 65 with fuzz 2 (offset 9 lines). Hunk #5 succeeded at 80 (offset 9 lines). Hunk #6 succeeded at 119 (offset 9 lines). Hunk #7 succeeded at 161 (offset 9 lines). Hunk #8 succeeded at 548 (offset 10 lines). Hunk #9 FAILED at 633. 3 out of 9 hunks FAILED -- saving rejects to file hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java.rej ... Could we keep ScheduledRequests as an inner class of RMContainerAllocator, i.e. change it to public/protected/package-private static? It will be easier to see the changes.
          Junping Du made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Junping Du made changes -
          Link This issue depends on HADOOP-8469 [ HADOOP-8469 ]
          Junping Du made changes -
          Link This issue blocks MAPREDUCE-4310 [ MAPREDUCE-4310 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2436//console This message is automatically generated.
          Junping Du made changes -
          Link This issue is part of HADOOP-8468 [ HADOOP-8468 ]
          Junping Du made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-8474 MAPREDUCE-4309
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Affects Version/s 1.0.0 [ 12318240 ]
          Affects Version/s 1.0.0 [ 12318244 ]
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Junping Du made changes -
          Parent HADOOP-8468 [ 12559114 ]
          Issue Type Sub-task [ 7 ] Bug [ 1 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530745/HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1078//console This message is automatically generated.
          Junping Du made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Junping Du made changes -
          Junping Du made changes -
          Field Original Value New Value
          Assignee Junping Du [ djp ]
          Junping Du created issue -

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:

                Development