Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    1. YARN-5756.1.patch
      21 kB
      Xuan Gong
    2. YARN-5756.2.patch
      27 kB
      Xuan Gong
    3. YARN-5756.3.patch
      32 kB
      Xuan Gong
    4. YARN-5756.4.patch
      34 kB
      Xuan Gong
    5. YARN-5756.5.patch
      35 kB
      Xuan Gong
    6. YARN-5756.6.patch
      38 kB
      Xuan Gong
    7. YARN-5756.6.patch
      40 kB
      Xuan Gong
    8. YARN-5756.7.patch
      38 kB
      Xuan Gong
    9. YARN-5756.8.patch
      44 kB
      Xuan Gong

      Issue Links

        Activity

        Hide
        xgong Xuan Gong added a comment -

        In this patch, created a QueueStateManager to manage the state for the queue.

        Show
        xgong Xuan Gong added a comment - In this patch, created a QueueStateManager to manage the state for the queue.
        Hide
        gtCarrera9 Li Lu added a comment -

        Hi Xuan Gong, I tried to apply the patch locally but there were several issues to apply to the latest trunk. One significant issue is SchedulerQueueContext.java is missing in trunk? Could you please rebase your patch? Thanks!

        Show
        gtCarrera9 Li Lu added a comment - Hi Xuan Gong , I tried to apply the patch locally but there were several issues to apply to the latest trunk. One significant issue is SchedulerQueueContext.java is missing in trunk? Could you please rebase your patch? Thanks!
        Hide
        xgong Xuan Gong added a comment -

        rebase the patch.

        Show
        xgong Xuan Gong added a comment - rebase the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        -1 patch 0m 4s YARN-5756 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837807/YARN-5756.1.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14137/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. -1 patch 0m 4s YARN-5756 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837807/YARN-5756.1.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/14137/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gtCarrera9 Li Lu added a comment -

        Seems like the second submission has been ignored by Jenkins. Kick it for one more round of testing.

        Show
        gtCarrera9 Li Lu added a comment - Seems like the second submission has been ignored by Jenkins. Kick it for one more round of testing.
        Hide
        gtCarrera9 Li Lu added a comment -

        Thanks Xuan Gong for the patch! Generally fine, some comments:

        QueueState.java

        • STOP_RUNNING state is a little bit confusing? How about RUNNING, CLOSED (or DRAINING), and STOPPED?
        • Java doc inconsistencies: at the very beginning of the enum we said there are only two possible states?

        QueueStateManager.java

        • Consistency issues with stop and activate queue? We're using fine grained locking to change each queue's status. We need to make the process of stopping each queue and its subqueues atomic (as in concurrency, not in db). Otherwise, concurrent activate queue calls may result in inconsistent results. If coarse grained locking is fine with the current use case, we may want to make activateQueues and stopQueues synchronized?
        • QueueStateManager only needs the queue mapping in SchedulerQueueManager, so we do not need to reference the whole SchedulerQueueManager here? I don't have a strong opinion here though...
        Show
        gtCarrera9 Li Lu added a comment - Thanks Xuan Gong for the patch! Generally fine, some comments: QueueState.java STOP_RUNNING state is a little bit confusing? How about RUNNING, CLOSED (or DRAINING), and STOPPED? Java doc inconsistencies: at the very beginning of the enum we said there are only two possible states? QueueStateManager.java Consistency issues with stop and activate queue? We're using fine grained locking to change each queue's status. We need to make the process of stopping each queue and its subqueues atomic (as in concurrency, not in db). Otherwise, concurrent activate queue calls may result in inconsistent results. If coarse grained locking is fine with the current use case, we may want to make activateQueues and stopQueues synchronized? QueueStateManager only needs the queue mapping in SchedulerQueueManager, so we do not need to reference the whole SchedulerQueueManager here? I don't have a strong opinion here though...
        Hide
        xgong Xuan Gong added a comment -

        Thanks for the review. Li Lu

        uploaded a new patch to address all your comments

        Show
        xgong Xuan Gong added a comment - Thanks for the review. Li Lu uploaded a new patch to address all your comments
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 55s Maven dependency ordering for branch
        +1 mvninstall 8m 15s trunk passed
        +1 compile 5m 35s trunk passed
        +1 checkstyle 0m 50s trunk passed
        +1 mvnsite 1m 30s trunk passed
        +1 mvneclipse 0m 42s trunk passed
        +1 findbugs 2m 16s trunk passed
        +1 javadoc 0m 53s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 58s the patch passed
        +1 compile 4m 52s the patch passed
        +1 javac 4m 52s the patch passed
        -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 236 unchanged - 2 fixed = 245 total (was 238)
        +1 mvnsite 1m 15s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 34s the patch passed
        +1 javadoc 0m 50s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        -1 unit 50m 46s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 34s The patch does not generate ASF License warnings.
        93m 17s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
          hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841798/YARN-5756.4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 47f665eaf0b2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8c46808
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14186/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14186/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14186/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14186/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 55s Maven dependency ordering for branch +1 mvninstall 8m 15s trunk passed +1 compile 5m 35s trunk passed +1 checkstyle 0m 50s trunk passed +1 mvnsite 1m 30s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 16s trunk passed +1 javadoc 0m 53s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 58s the patch passed +1 compile 4m 52s the patch passed +1 javac 4m 52s the patch passed -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 9 new + 236 unchanged - 2 fixed = 245 total (was 238) +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 39s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 34s the patch passed +1 javadoc 0m 50s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. -1 unit 50m 46s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 93m 17s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue   hadoop.yarn.server.resourcemanager.reservation.TestCapacitySchedulerPlanFollower   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerDynamicBehavior   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841798/YARN-5756.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 47f665eaf0b2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8c46808 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14186/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14186/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14186/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14186/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gtCarrera9 Li Lu added a comment -

        Thanks Xuan Gong. Looks fine but I'm not extremely familiar with queue/schedulers. Maybe Tan, Wangda or Jian He can take a look at it?

        Show
        gtCarrera9 Li Lu added a comment - Thanks Xuan Gong . Looks fine but I'm not extremely familiar with queue/schedulers. Maybe Tan, Wangda or Jian He can take a look at it?
        Hide
        xgong Xuan Gong added a comment -

        Fix the test-case failures

        Show
        xgong Xuan Gong added a comment - Fix the test-case failures
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 7m 46s trunk passed
        +1 compile 5m 13s trunk passed
        +1 checkstyle 0m 54s trunk passed
        +1 mvnsite 1m 20s trunk passed
        +1 mvneclipse 0m 42s trunk passed
        +1 findbugs 2m 17s trunk passed
        +1 javadoc 0m 51s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 56s the patch passed
        +1 compile 4m 51s the patch passed
        +1 javac 4m 51s the patch passed
        -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 609 unchanged - 3 fixed = 611 total (was 612)
        +1 mvnsite 1m 20s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 27s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        -1 unit 42m 44s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        83m 39s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841852/YARN-5756.5.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2b3a1e8e885f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dcedb72
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14191/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14191/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 7m 46s trunk passed +1 compile 5m 13s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 2m 17s trunk passed +1 javadoc 0m 51s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 56s the patch passed +1 compile 4m 51s the patch passed +1 javac 4m 51s the patch passed -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 609 unchanged - 3 fixed = 611 total (was 612) +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 27s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 0m 32s hadoop-yarn-api in the patch passed. -1 unit 42m 44s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 83m 39s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueState Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12841852/YARN-5756.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2b3a1e8e885f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dcedb72 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14191/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14191/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14191/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Xuan Gong, some comments/questions.

        General:

        • Could you add some descriptions to YARN-5755 and YARN-5724 as mentioned by QueueStateManager

        1) QueueStateManager:

        • queueContext -> queueManager
        • Move AbstractCSQueue#initializeQueueState to QueueStateManager? (Which we can protect all state transitions under the same sync lock)
          appFinished needs sync lock.

        2) AbstractCSQueue

          private ReentrantReadWriteLock.ReadLock queueStateReadLock;
          private ReentrantReadWriteLock.WriteLock queueStateWriteLock;
        

        Is not necessary since state is volatile

        3) Implementation of initializeQueueState

        • I'm not sure if initializeQueueState should be called when admin refreshes queue config. If we plan to do so, initializeQueueState should be invoked when reinitialize the queue, so you may need to move it to setupQueueConfigs.
        • And in addition, if initializeQueueState will be called when admin refreshes queue config, we need to handle state-in-memory and state-from-config, for example, draining state.
        Show
        leftnoteasy Wangda Tan added a comment - Thanks Xuan Gong , some comments/questions. General: Could you add some descriptions to YARN-5755 and YARN-5724 as mentioned by QueueStateManager 1) QueueStateManager: queueContext -> queueManager Move AbstractCSQueue#initializeQueueState to QueueStateManager? (Which we can protect all state transitions under the same sync lock) appFinished needs sync lock. 2) AbstractCSQueue private ReentrantReadWriteLock.ReadLock queueStateReadLock; private ReentrantReadWriteLock.WriteLock queueStateWriteLock; Is not necessary since state is volatile 3) Implementation of initializeQueueState I'm not sure if initializeQueueState should be called when admin refreshes queue config. If we plan to do so, initializeQueueState should be invoked when reinitialize the queue, so you may need to move it to setupQueueConfigs . And in addition, if initializeQueueState will be called when admin refreshes queue config, we need to handle state-in-memory and state-from-config, for example, draining state.
        Hide
        xgong Xuan Gong added a comment -

        Thanks for the comments, Wangda Tan

        Have uploaded a new patch to address the comments

        Show
        xgong Xuan Gong added a comment - Thanks for the comments, Wangda Tan Have uploaded a new patch to address the comments
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Xuan Gong,

        Generally looks good, some minor comments:

        1) AbstractCSQueue#initializeQueueState:

        • initializeQueueState: is it better to rename current to previous?
        • And can we make the initializeQueueState accept 3 parameters: previous/configured/parent, with this we can do a more easier and comprehensive UT to make sure it won't break in the future.

        2) Leaf(Parent)Queue#haveActiveApps can move to abstractCSQueue, or just completely remove the method since we can use getNumApplications instead.

        3) AbstractCSQueue#updateQueueState, writeLock is not necessary for volatile. queueStateWriteLock can be directly removed.

        4) QueueStateManager is not used by anybody now? Is it better to move to a separated patch? Which JIRA you plan to use it?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Xuan Gong , Generally looks good, some minor comments: 1) AbstractCSQueue#initializeQueueState: initializeQueueState: is it better to rename current to previous? And can we make the initializeQueueState accept 3 parameters: previous/configured/parent, with this we can do a more easier and comprehensive UT to make sure it won't break in the future. 2) Leaf(Parent)Queue#haveActiveApps can move to abstractCSQueue, or just completely remove the method since we can use getNumApplications instead. 3) AbstractCSQueue#updateQueueState, writeLock is not necessary for volatile. queueStateWriteLock can be directly removed. 4) QueueStateManager is not used by anybody now? Is it better to move to a separated patch? Which JIRA you plan to use it?
        Hide
        xgong Xuan Gong added a comment -

        Tan, Wangda Thanks for the view.

        Attached a patch for all the comments

        Show
        xgong Xuan Gong added a comment - Tan, Wangda Thanks for the view. Attached a patch for all the comments
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 12m 42s trunk passed
        +1 compile 5m 0s trunk passed
        +1 checkstyle 0m 51s trunk passed
        +1 mvnsite 1m 13s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 2m 15s trunk passed
        +1 javadoc 0m 51s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 0m 59s the patch passed
        +1 compile 4m 37s the patch passed
        +1 cc 4m 37s the patch passed
        +1 javac 4m 37s the patch passed
        -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 610 unchanged - 3 fixed = 613 total (was 613)
        +1 mvnsite 1m 12s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 32s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 0m 30s hadoop-yarn-api in the patch passed.
        +1 unit 42m 38s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        87m 25s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843662/YARN-5756.7.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 7c317c1fe731 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f121645
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14351/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14351/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14351/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 12m 42s trunk passed +1 compile 5m 0s trunk passed +1 checkstyle 0m 51s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 15s trunk passed +1 javadoc 0m 51s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 59s the patch passed +1 compile 4m 37s the patch passed +1 cc 4m 37s the patch passed +1 javac 4m 37s the patch passed -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 610 unchanged - 3 fixed = 613 total (was 613) +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 32s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 0m 30s hadoop-yarn-api in the patch passed. +1 unit 42m 38s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 87m 25s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843662/YARN-5756.7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 7c317c1fe731 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f121645 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14351/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14351/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14351/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Xuan Gong, generally looks good, only one comment, could you add some test cases to make sure initializeQueueState works well?
        I'm afraid regression could easily happen for any change to the method.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Xuan Gong , generally looks good, only one comment, could you add some test cases to make sure initializeQueueState works well? I'm afraid regression could easily happen for any change to the method.
        Hide
        xgong Xuan Gong added a comment -

        Added a new testcase for this.

        Please review.

        Show
        xgong Xuan Gong added a comment - Added a new testcase for this. Please review.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 13m 52s trunk passed
        +1 compile 5m 29s trunk passed
        +1 checkstyle 0m 55s trunk passed
        +1 mvnsite 1m 20s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 2m 27s trunk passed
        +1 javadoc 0m 54s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 4s the patch passed
        +1 compile 5m 13s the patch passed
        +1 cc 5m 13s the patch passed
        +1 javac 5m 13s the patch passed
        -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 610 unchanged - 3 fixed = 614 total (was 613)
        +1 mvnsite 1m 22s the patch passed
        +1 mvneclipse 0m 39s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 2m 45s the patch passed
        +1 javadoc 0m 50s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        -1 unit 39m 43s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        88m 4s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5756
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844321/YARN-5756.8.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc
        uname Linux 5f643b35a520 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 736f54b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14434/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14434/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 13m 52s trunk passed +1 compile 5m 29s trunk passed +1 checkstyle 0m 55s trunk passed +1 mvnsite 1m 20s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 27s trunk passed +1 javadoc 0m 54s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 5m 13s the patch passed +1 cc 5m 13s the patch passed +1 javac 5m 13s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 610 unchanged - 3 fixed = 614 total (was 613) +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 39s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 2m 45s the patch passed +1 javadoc 0m 50s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. -1 unit 39m 43s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 88m 4s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5756 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844321/YARN-5756.8.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc uname Linux 5f643b35a520 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 736f54b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14434/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14434/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14434/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        +1, will commit tomorrow if no opposite opinions. Thanks Xuan Gong

        Show
        leftnoteasy Wangda Tan added a comment - +1, will commit tomorrow if no opposite opinions. Thanks Xuan Gong
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk / branch-2, thanks Xuan Gong for working on this and thanks reviews from Li Lu!

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk / branch-2, thanks Xuan Gong for working on this and thanks reviews from Li Lu !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11046 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11046/)
        YARN-5756. Add state-machine implementation for scheduler queues. (Xuan (wangda: rev 0840b4329b2428b20b862f70d72cbdcd6d1618ed)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueStateManager.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueStateManager.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerQueueManager.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11046 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11046/ ) YARN-5756 . Add state-machine implementation for scheduler queues. (Xuan (wangda: rev 0840b4329b2428b20b862f70d72cbdcd6d1618ed) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueStateManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueStateManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/QueueState.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerQueueManager.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java

          People

          • Assignee:
            xgong Xuan Gong
            Reporter:
            xgong Xuan Gong
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development