Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: scheduler
    • Labels:
      None

      Description

      Currently, both of fair/capacity scheduler support queue mapping, which makes scheduler can change queue of an application after submitted to scheduler.

      One issue of doing this in specific scheduler is: If the queue after mapping has different maximum_allocation/default-node-label-expression of the original queue, validateAndCreateResourceRequest in RMAppManager checks the wrong queue.

      I propose to make the queue mapping as a common interface of scheduler, and RMAppManager set the queue after mapping before doing validations.

      1. YARN-3635.1.patch
        12 kB
        Wangda Tan
      2. YARN-3635.2.patch
        14 kB
        Wangda Tan
      3. YARN-3635.3.patch
        14 kB
        Wangda Tan
      4. YARN-3635.4.patch
        45 kB
        Wangda Tan
      5. YARN-3635.5.patch
        48 kB
        Wangda Tan
      6. YARN-3635.6.patch
        50 kB
        Wangda Tan
      7. YARN-3635.7.patch
        50 kB
        Wangda Tan
      8. YARN-3635.8.patch
        50 kB
        Wangda Tan

        Activity

        Show
        leftnoteasy Wangda Tan added a comment - + Vinod Kumar Vavilapalli , Karthik Kambatla , Jason Lowe , Jian He
        Hide
        kasha Karthik Kambatla added a comment -

        I am in favor of doing this, but would like to be extra careful. Can we make sure either Sandy Ryza or I get to review this before it is committed.

        Show
        kasha Karthik Kambatla added a comment - I am in favor of doing this, but would like to be extra careful. Can we make sure either Sandy Ryza or I get to review this before it is committed.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        +10 for this.

        In addition to the interfaces themselves, I'd like us to consolidate the concrete mapping-rules that we have in each scheduler. Ideally, we only need one set of rules acceptable by all schedulers. If not that, I'd live with ~80% common rules.

        BTW, thematically this fits into YARN-1317, making it a sub-task.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - +10 for this. In addition to the interfaces themselves, I'd like us to consolidate the concrete mapping-rules that we have in each scheduler. Ideally, we only need one set of rules acceptable by all schedulers. If not that, I'd live with ~80% common rules. BTW, thematically this fits into YARN-1317 , making it a sub-task.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Karthik Kambatla, Thanks for supporting this. Sure I will ping you once I get a patch.
        Vinod Kumar Vavilapalli, I will take a look at implementation in different schedulers, I will focus on interface itself in the patch first, and to see we can address the common logic merging together in the patch.

        Show
        leftnoteasy Wangda Tan added a comment - Karthik Kambatla , Thanks for supporting this. Sure I will ping you once I get a patch. Vinod Kumar Vavilapalli , I will take a look at implementation in different schedulers, I will focus on interface itself in the patch first, and to see we can address the common logic merging together in the patch.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.1 patch. With changes of RMAppManager, added interface in YarnScheduler and updated CapacityScheduler to use the interface.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.1 patch. With changes of RMAppManager, added interface in YarnScheduler and updated CapacityScheduler to use the interface.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 18m 7s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
        +1 javac 9m 9s There were no new javac warning messages.
        +1 javadoc 12m 9s There were no new javadoc warning messages.
        +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 55s The applied patch generated 1 new checkstyle issues (total was 210, now 211).
        -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 46s mvn install still works.
        +1 eclipse:eclipse 0m 57s The patch built with eclipse:eclipse.
        +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        -1 yarn tests 46m 29s Tests failed in hadoop-yarn-server-resourcemanager.
            91m 55s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestClientRMService
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732630/YARN-3635.1.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 065d8f2
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7923/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7923/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 7s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 9m 9s There were no new javac warning messages. +1 javadoc 12m 9s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 55s The applied patch generated 1 new checkstyle issues (total was 210, now 211). -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 46s mvn install still works. +1 eclipse:eclipse 0m 57s The patch built with eclipse:eclipse. +1 findbugs 1m 53s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 46m 29s Tests failed in hadoop-yarn-server-resourcemanager.     91m 55s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestClientRMService Timed out tests org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732630/YARN-3635.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 065d8f2 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7923/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7923/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7923/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Fixed test failure (ver.2)

        Show
        leftnoteasy Wangda Tan added a comment - Fixed test failure (ver.2)
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 34s There were no new javac warning messages.
        +1 javadoc 9m 36s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 45s The applied patch generated 1 new checkstyle issues (total was 210, now 211).
        -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 34s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        -1 yarn tests 49m 51s Tests failed in hadoop-yarn-server-resourcemanager.
            86m 12s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732726/YARN-3635.2.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0e85044
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7929/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7929/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 36s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 45s The applied patch generated 1 new checkstyle issues (total was 210, now 211). -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 14s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 49m 51s Tests failed in hadoop-yarn-server-resourcemanager.     86m 12s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732726/YARN-3635.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0e85044 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7929/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7929/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7929/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Failed test is not related to this change, which is tracked by YARN-3504

        Show
        leftnoteasy Wangda Tan added a comment - Failed test is not related to this change, which is tracked by YARN-3504
        Hide
        jianhe Jian He added a comment -

        looks good overall, getQueueNameOfNewApp, may be call it getMappedQueue ?

        Show
        jianhe Jian He added a comment - looks good overall, getQueueNameOfNewApp, may be call it getMappedQueue ?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Jian He's review. Uploaded ver.3 patch with modified name.

        And forgot to mention that, I tried to update FairScheduler with the new interface (which uses QueuePlacementPolicy's result). It fits in the interface quite well, but since FS depends on a lot of mocking mechanism, needs some time to make changes in the tests, so I think it's better to do in a separate JIRA.

        Karthik Kambatla, does this make sense to you, could you take a look at latest patch?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Jian He 's review. Uploaded ver.3 patch with modified name. And forgot to mention that, I tried to update FairScheduler with the new interface (which uses QueuePlacementPolicy's result). It fits in the interface quite well, but since FS depends on a lot of mocking mechanism, needs some time to make changes in the tests, so I think it's better to do in a separate JIRA. Karthik Kambatla , does this make sense to you, could you take a look at latest patch?
        Hide
        kasha Karthik Kambatla added a comment -

        With FairScheduler, if this patch is applied, does the QueuePlacementPolicy cease to work?

        Show
        kasha Karthik Kambatla added a comment - With FairScheduler, if this patch is applied, does the QueuePlacementPolicy cease to work?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
        +1 javac 7m 29s There were no new javac warning messages.
        +1 javadoc 9m 33s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 48s The applied patch generated 1 new checkstyle issues (total was 210, now 211).
        -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 findbugs 1m 18s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.
        +1 yarn tests 50m 2s Tests passed in hadoop-yarn-server-resourcemanager.
            86m 11s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
          Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.isHDFS; locked 66% of time Unsynchronized access at FileSystemRMStateStore.java:66% of time Unsynchronized access at FileSystemRMStateStore.java:[line 156]



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733003/YARN-3635.3.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9a2a955
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7945/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7945/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 29s There were no new javac warning messages. +1 javadoc 9m 33s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 48s The applied patch generated 1 new checkstyle issues (total was 210, now 211). -1 whitespace 0m 1s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 1m 18s The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 yarn tests 50m 2s Tests passed in hadoop-yarn-server-resourcemanager.     86m 11s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager   Inconsistent synchronization of org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore.isHDFS; locked 66% of time Unsynchronized access at FileSystemRMStateStore.java:66% of time Unsynchronized access at FileSystemRMStateStore.java: [line 156] Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733003/YARN-3635.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9a2a955 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7945/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7945/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7945/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Karthik Kambatla,
        Actually no, this patch doesn't do anything to RM when using FS, RMAppManager use the original user specified queue name to do initialization, and FS can continue applying QueuePlacementPolicy after app added to scheduler. FS tests can all get passed also.

        Any other thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Karthik Kambatla , Actually no, this patch doesn't do anything to RM when using FS, RMAppManager use the original user specified queue name to do initialization, and FS can continue applying QueuePlacementPolicy after app added to scheduler. FS tests can all get passed also. Any other thoughts?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Okay, I didn't think you were really meaning the JIRA title literally.

        In general, I think we should have a separate component sitting next to ClientService responsible for mapping apps to queues. I've seen lot more use-cases beyond what we have today in either of the schedulers - it will be useful to start laying this out with an ultimate intention to potentially make it an end-user visible abstraction.

        Here's what I propose

        • PlacementRule represents a rule with the following interface: PlacementAction matches(Application). Examples includes matching queue-name argument, user-name to queue mapping etc.
        • PlacementAction represents the action to take: simplest is to submit to a queue.
        • PlacementRuleChain: A chain of PlacementRules
        • PlacementRuleChain getting initialized from scheduler specific config files for now
        • A PlacementManager which keeps track of PlacementRuleChain and takes action on incoming apps

        Thoughts?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Okay, I didn't think you were really meaning the JIRA title literally. In general, I think we should have a separate component sitting next to ClientService responsible for mapping apps to queues. I've seen lot more use-cases beyond what we have today in either of the schedulers - it will be useful to start laying this out with an ultimate intention to potentially make it an end-user visible abstraction. Here's what I propose PlacementRule represents a rule with the following interface: PlacementAction matches(Application) . Examples includes matching queue-name argument, user-name to queue mapping etc. PlacementAction represents the action to take: simplest is to submit to a queue. PlacementRuleChain: A chain of PlacementRules PlacementRuleChain getting initialized from scheduler specific config files for now A PlacementManager which keeps track of PlacementRuleChain and takes action on incoming apps Thoughts?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sorry for my late response, Vinod Kumar Vavilapalli. Just have some bandwidth to do the update.

        Attached ver.4 addressed most of your comments, now queue-placement-rules is a separated module in RM, and scheduler initializes it. RMAppManager uses it to do queue placing.

        Defined interfaces are not exactly as same as you suggested, I put minimal set of interfaces needed in my mind. You can take a look at: org.apache.hadoop.yarn.server.resourcemanager.placement for details.

        And the ver.4 patch makes original CapacityScheduler.QueueMapping becomes a rule: UserGroupPlacementRule.

        Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Sorry for my late response, Vinod Kumar Vavilapalli . Just have some bandwidth to do the update. Attached ver.4 addressed most of your comments, now queue-placement-rules is a separated module in RM, and scheduler initializes it. RMAppManager uses it to do queue placing. Defined interfaces are not exactly as same as you suggested, I put minimal set of interfaces needed in my mind. You can take a look at: org.apache.hadoop.yarn.server.resourcemanager.placement for details. And the ver.4 patch makes original CapacityScheduler.QueueMapping becomes a rule: UserGroupPlacementRule. Thoughts?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 14s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 43s There were no new javac warning messages.
        -1 javadoc 9m 48s The applied patch generated 2 additional warning messages.
        -1 release audit 0m 18s The applied patch generated 4 release audit warnings.
        -1 checkstyle 0m 47s The applied patch generated 18 new checkstyle issues (total was 204, now 215).
        -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 32s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 1m 27s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 61m 8s Tests failed in hadoop-yarn-server-resourcemanager.
            99m 39s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741096/YARN-3635.4.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 445b132
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffJavadocWarnings.txt
        Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/patchReleaseAuditProblems.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8311/testReport/
        Java 1.7.0_55
        uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8311/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. -1 javadoc 9m 48s The applied patch generated 2 additional warning messages. -1 release audit 0m 18s The applied patch generated 4 release audit warnings. -1 checkstyle 0m 47s The applied patch generated 18 new checkstyle issues (total was 204, now 215). -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 1m 27s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 yarn tests 61m 8s Tests failed in hadoop-yarn-server-resourcemanager.     99m 39s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741096/YARN-3635.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 445b132 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffJavadocWarnings.txt Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/patchReleaseAuditProblems.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8311/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8311/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8311/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.5, fixed bunch of warnings.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.5, fixed bunch of warnings.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 25s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 55s There were no new javac warning messages.
        +1 javadoc 10m 0s There were no new javadoc warning messages.
        +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 46s The applied patch generated 18 new checkstyle issues (total was 204, now 215).
        -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 40s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 1m 30s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 50m 14s Tests failed in hadoop-yarn-server-resourcemanager.
            89m 34s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager
        Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 077250d
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8316/testReport/
        Java 1.7.0_55
        uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8316/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 25s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 46s The applied patch generated 18 new checkstyle issues (total was 204, now 215). -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 40s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 1m 30s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. -1 yarn tests 50m 14s Tests failed in hadoop-yarn-server-resourcemanager.     89m 34s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Failed unit tests hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 077250d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8316/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8316/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8316/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 17m 9s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 8m 10s There were no new javac warning messages.
        +1 javadoc 9m 59s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 52s The applied patch generated 18 new checkstyle issues (total was 210, now 221).
        -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 25s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        -1 findbugs 1m 32s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 51m 31s Tests passed in hadoop-yarn-server-resourcemanager.
            91m 40s  



        Reason Tests
        FindBugs module:hadoop-yarn-server-resourcemanager



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 9ef03a4
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/whitespace.txt
        Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8525/testReport/
        Java 1.7.0_55
        uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8525/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 10s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 52s The applied patch generated 18 new checkstyle issues (total was 210, now 221). -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 1m 32s The patch appears to introduce 3 new Findbugs (version 3.0.0) warnings. +1 yarn tests 51m 31s Tests passed in hadoop-yarn-server-resourcemanager.     91m 40s   Reason Tests FindBugs module:hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12741145/YARN-3635.5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 9ef03a4 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8525/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8525/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8525/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        lgtm overall,

        • the queuePlacementManager should be added in RMActiveServicesContext which includes all active services.
            private PlacementManager queuePlacementManager;
          
        Show
        jianhe Jian He added a comment - lgtm overall, the queuePlacementManager should be added in RMActiveServicesContext which includes all active services. private PlacementManager queuePlacementManager;
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for comment, Jian He. Uploaded new patch moving queue placement manager from RMContext to RMActiveContext and fixed findbugs warnings.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for comment, Jian He . Uploaded new patch moving queue placement manager from RMContext to RMActiveContext and fixed findbugs warnings.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 15s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 40s There were no new javac warning messages.
        +1 javadoc 9m 40s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 49s The applied patch generated 14 new checkstyle issues (total was 234, now 241).
        -1 whitespace 0m 4s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 21s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 51m 7s Tests passed in hadoop-yarn-server-resourcemanager.
            89m 23s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a431ed9
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8526/testReport/
        Java 1.7.0_55
        uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8526/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 15s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 49s The applied patch generated 14 new checkstyle issues (total was 234, now 241). -1 whitespace 0m 4s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 26s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 51m 7s Tests passed in hadoop-yarn-server-resourcemanager.     89m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a431ed9 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8526/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8526/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8526/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        newest patch looks good to me, + !

        Show
        jianhe Jian He added a comment - newest patch looks good to me, + !
        Hide
        leftnoteasy Wangda Tan added a comment -

        Karthik Kambatla/Sandy Ryza, do you want to take a look at latest patch?

        Show
        leftnoteasy Wangda Tan added a comment - Karthik Kambatla / Sandy Ryza , do you want to take a look at latest patch?
        Hide
        sandyr Sandy Ryza added a comment -

        Wangda Tan, apologies for this quick drive-by review - I am currently traveling.

        The JIRA appears to be lacking a design-doc and I wasn't able to find documentation in the patch. The patch should ultimately include some detailed documentation, but I don't want to ask this of you before OKing the approach. In light of this, a few questions:

        • What steps are required for the Fair Scheduler to integrate with this?
        • Is a common way of configuration proposed?
        • How does this differ from the current Fair Scheduler model? To summarize:
          • The FS model consists of a sequence of placement rules that the app is passed through.
          • Each placement rule gets the chance to assign the app to a queue, reject the app, or pass. If it passes, the next rule gets a chance.
          • A placement rule can base its decision on:
            • The submitting user.
            • The set of groups the submitting user belongs to.
            • The queue requested in the app submission.
            • A set of configuration options that are specific to the rule.
            • The set of queues given in the Fair Scheduler configuration.
          • Rules are marked as "terminal" if they will never pass. This helps to avoid misconfigurations where users place rules after terminal rules.
          • Rules have a "create" attribute which determines whether they can create a new queue or whether they must assign to existing queues.
          • Currently the set of placement rules is limited to what's implemented in YARN. I.e. there's no public pluggable rule support.
            I noticed from Vinod's comment that this patch follows a similar structure. Are there places where my summary would not describe what's going on in this patch?
        Show
        sandyr Sandy Ryza added a comment - Wangda Tan , apologies for this quick drive-by review - I am currently traveling. The JIRA appears to be lacking a design-doc and I wasn't able to find documentation in the patch. The patch should ultimately include some detailed documentation, but I don't want to ask this of you before OKing the approach. In light of this, a few questions: What steps are required for the Fair Scheduler to integrate with this? Is a common way of configuration proposed? How does this differ from the current Fair Scheduler model? To summarize: The FS model consists of a sequence of placement rules that the app is passed through. Each placement rule gets the chance to assign the app to a queue, reject the app, or pass. If it passes, the next rule gets a chance. A placement rule can base its decision on: The submitting user. The set of groups the submitting user belongs to. The queue requested in the app submission. A set of configuration options that are specific to the rule. The set of queues given in the Fair Scheduler configuration. Rules are marked as "terminal" if they will never pass. This helps to avoid misconfigurations where users place rules after terminal rules. Rules have a "create" attribute which determines whether they can create a new queue or whether they must assign to existing queues. Currently the set of placement rules is limited to what's implemented in YARN. I.e. there's no public pluggable rule support. I noticed from Vinod's comment that this patch follows a similar structure. Are there places where my summary would not describe what's going on in this patch?
        Hide
        sandyr Sandy Ryza added a comment -

        BTW I got all this from QueuePlacementPolicy and QueuePlacementRule, which are pretty quick reads if you want to take a look.

        Show
        sandyr Sandy Ryza added a comment - BTW I got all this from QueuePlacementPolicy and QueuePlacementRule, which are pretty quick reads if you want to take a look.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Hi Sandy Ryza,
        Thanks for your comments, actually I have read QueuePlacementPolicy/QueuePlacementRule from FS before working on this patch. The generic design fo this patch is based on FS's queue placement policy structure, but also with some changes.

        To your comments:

        Is a common way of configuration proposed?

        No common configuration, it only defined a set of common interfaces. Since FS/CS have very different ways to configuration, now rules are created by different schedulers, see CapacityScheduler#updatePlacementRules as an example.

        What steps are required for the Fair Scheduler to integrate with this?

        1) Port existing rules to new APIs defined in the patch, this should be simple
        2) Change configuration implementation to instance new defined PlacementRule, you may not need to change existing configuration items itself.
        3) Change FS workflow, with this patch, queue mapping is happened before submit to scheduler. Remove queue mapping related logics from FS and create queue if needed.

        Each placement rule gets the chance to assign the app to a queue, reject the app, or pass. If it passes, the next rule gets a chance.

        New APIs are very similar:
        Non-null is determined
        Null is not determined
        Throw exception when rejected.
        You can take a look at org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementRule

        A placement rule can base its decision on:

        ....

        Yes you can do all of them with the new API except "The set of queues given in the Fair Scheduler configuration.":
        I was thinking necessarity of passing set of queues in the interface. In existing implementations of QueuePlacementPolicy, FS queues are only used to check mapped queue's existence. I would prefer to delay the check to submit to scheduler. See my next comment about "create" flag for more details.
        Another reason of not passing queue names set via interface is, queues are very dynamic. For example, if user wants to submit application to queue with lowest utilization, queue names set may not enough. I would prefer to let rule choose to get what need from scheduler.

        Rules are marked as "terminal" if they will never pass. This helps to avoid misconfigurations where users place rules after terminal rules.

        I'm not sure if is it necessary. I think terminal or not should be determined by runtime, but I'm OK if you think it's must to have.

        Rules have a "create" attribute which determines whether they can create a new queue or whether they must assign to existing queues.

        I think queue is create-able or not should be determined by scheduler, it should be a part of scheduler configuration instead of rule itself.
        You can put "create" to your implemented rules without any issue, but I prefer not to expose it to public interface.

        Currently the set of placement rules is limited to what's implemented in YARN. I.e. there's no public pluggable rule support.

        Agree, this is one thing we need to do in the future. For now, we can make queue mapping happens in a central place first.

        Are there places where my summary would not describe what's going on in this patch?

        I think it should covers most of my patch, you can also take a look at my patch to see if anything unexpected .

        Show
        leftnoteasy Wangda Tan added a comment - Hi Sandy Ryza , Thanks for your comments, actually I have read QueuePlacementPolicy/QueuePlacementRule from FS before working on this patch. The generic design fo this patch is based on FS's queue placement policy structure, but also with some changes. To your comments: Is a common way of configuration proposed? No common configuration, it only defined a set of common interfaces. Since FS/CS have very different ways to configuration, now rules are created by different schedulers, see CapacityScheduler#updatePlacementRules as an example. What steps are required for the Fair Scheduler to integrate with this? 1) Port existing rules to new APIs defined in the patch, this should be simple 2) Change configuration implementation to instance new defined PlacementRule, you may not need to change existing configuration items itself. 3) Change FS workflow, with this patch, queue mapping is happened before submit to scheduler. Remove queue mapping related logics from FS and create queue if needed. Each placement rule gets the chance to assign the app to a queue, reject the app, or pass. If it passes, the next rule gets a chance. New APIs are very similar: Non-null is determined Null is not determined Throw exception when rejected. You can take a look at org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementRule A placement rule can base its decision on: .... Yes you can do all of them with the new API except "The set of queues given in the Fair Scheduler configuration.": I was thinking necessarity of passing set of queues in the interface. In existing implementations of QueuePlacementPolicy, FS queues are only used to check mapped queue's existence. I would prefer to delay the check to submit to scheduler. See my next comment about "create" flag for more details. Another reason of not passing queue names set via interface is, queues are very dynamic. For example, if user wants to submit application to queue with lowest utilization, queue names set may not enough. I would prefer to let rule choose to get what need from scheduler. Rules are marked as "terminal" if they will never pass. This helps to avoid misconfigurations where users place rules after terminal rules. I'm not sure if is it necessary. I think terminal or not should be determined by runtime, but I'm OK if you think it's must to have. Rules have a "create" attribute which determines whether they can create a new queue or whether they must assign to existing queues. I think queue is create-able or not should be determined by scheduler, it should be a part of scheduler configuration instead of rule itself. You can put "create" to your implemented rules without any issue, but I prefer not to expose it to public interface. Currently the set of placement rules is limited to what's implemented in YARN. I.e. there's no public pluggable rule support. Agree, this is one thing we need to do in the future. For now, we can make queue mapping happens in a central place first. Are there places where my summary would not describe what's going on in this patch? I think it should covers most of my patch, you can also take a look at my patch to see if anything unexpected .
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sandy Ryza, any comments on reply above?

        Show
        leftnoteasy Wangda Tan added a comment - Sandy Ryza , any comments on reply above?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / a4d9acc
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8918/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12745158/YARN-3635.6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Console output https://builds.apache.org/job/PreCommit-YARN-Build/8918/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased to latest trunk (ver.7)

        Show
        leftnoteasy Wangda Tan added a comment - Rebased to latest trunk (ver.7)
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 40s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 51s There were no new javac warning messages.
        +1 javadoc 10m 8s There were no new javadoc warning messages.
        +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 52s The applied patch generated 14 new checkstyle issues (total was 236, now 242).
        -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 54m 13s Tests passed in hadoop-yarn-server-resourcemanager.
            93m 49s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12754296/YARN-3635.7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 970daaa
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9040/testReport/
        Java 1.7.0_55
        uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9040/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 javadoc 10m 8s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 52s The applied patch generated 14 new checkstyle issues (total was 236, now 242). -1 whitespace 0m 3s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 54m 13s Tests passed in hadoop-yarn-server-resourcemanager.     93m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754296/YARN-3635.7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 970daaa checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9040/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9040/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9040/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased to latest trunk (ver.8)

        Show
        leftnoteasy Wangda Tan added a comment - Rebased to latest trunk (ver.8)
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 16s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 7m 36s There were no new javac warning messages.
        +1 javadoc 9m 48s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 46s The applied patch generated 14 new checkstyle issues (total was 234, now 241).
        -1 whitespace 0m 6s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 25s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        -1 yarn tests 45m 40s Tests failed in hadoop-yarn-server-resourcemanager.
            84m 5s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
          hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
          hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions
          hadoop.yarn.server.resourcemanager.TestApplicationCleanup
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
          hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
          hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
          hadoop.yarn.server.resourcemanager.TestRM
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
          hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
          hadoop.yarn.server.resourcemanager.TestResourceManager
          hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
          hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 332b520
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9112/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9112/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 16s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 36s There were no new javac warning messages. +1 javadoc 9m 48s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 46s The applied patch generated 14 new checkstyle issues (total was 234, now 241). -1 whitespace 0m 6s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 25s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 yarn tests 45m 40s Tests failed in hadoop-yarn-server-resourcemanager.     84m 5s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens   hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService   hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens   hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions   hadoop.yarn.server.resourcemanager.TestApplicationCleanup   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels   hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher   hadoop.yarn.server.resourcemanager.TestRM   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps   hadoop.yarn.server.resourcemanager.rmapp.TestNodesListManager   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes   hadoop.yarn.server.resourcemanager.TestResourceManager   hadoop.yarn.server.resourcemanager.TestContainerResourceUsage   hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 332b520 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9112/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9112/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9112/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 16m 48s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 javac 8m 1s There were no new javac warning messages.
        +1 javadoc 10m 12s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 50s The applied patch generated 14 new checkstyle issues (total was 234, now 241).
        -1 whitespace 0m 6s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 31s mvn install still works.
        +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
        +1 findbugs 1m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
        +1 yarn tests 54m 31s Tests passed in hadoop-yarn-server-resourcemanager.
            94m 30s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / e1b1d7e
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9118/testReport/
        Java 1.7.0_55
        uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/9118/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 48s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 8m 1s There were no new javac warning messages. +1 javadoc 10m 12s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 50s The applied patch generated 14 new checkstyle issues (total was 234, now 241). -1 whitespace 0m 6s The patch has 15 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 1m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 54m 31s Tests passed in hadoop-yarn-server-resourcemanager.     94m 30s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12755647/YARN-3635.8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / e1b1d7e checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/whitespace.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9118/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9118/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9118/console This message was automatically generated.
        Hide
        jianhe Jian He added a comment -

        lgtm

        Show
        jianhe Jian He added a comment - lgtm
        Hide
        jianhe Jian He added a comment -

        +1, committing soon if no further comments.

        Show
        jianhe Jian He added a comment - +1, committing soon if no further comments.
        Hide
        jianhe Jian He added a comment -

        Committed to trunk and branch-2, thanks Wangda !

        Show
        jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Wangda !
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8453 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8453/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8453 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8453/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2312 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2312/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2312 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2312/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Yarn-trunk #1127 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1127/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Yarn-trunk #1127 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1127/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Mapreduce-trunk-Java8 #388 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/388/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Mapreduce-trunk-Java8 #388 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/388/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #372 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/372/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #372 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/372/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2336 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2336/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2336 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2336/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #394 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/394/)
        YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #394 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/394/ ) YARN-3635 . Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan (jianhe: rev 5468baa80aa2a3e2a02e9a902deebafd734daf23) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java

          People

          • Assignee:
            wangda Tan, Wangda
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development