Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6151

FS preemption does not consider child queues over fairshare if the parent is under

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This is preemption bug happens before 2.8.0, which also described in YARN-3405.

      Queue hierarchy described as below:

                        root
                     /         \
             queue-1          queue-2       
                /      \
      queue-1-1     queue-1-2
      

      Assume cluster resource is 100 and all queues have same weights.

      1. queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare.
      2. When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but this doesn't happen because preemption happens top-down, queue-2 could be the preemption candidate as long as queue-2 is less needy than queue-1, and queue-2 doesn't exceed the fair share which means preemption won't happen.

      We need to filter out queue-2 since it isn't a valid candidate.

      1. YARN-6151.branch-2.8.001.patch
        6 kB
        Yufei Gu
      2. YARN-6151.branch-2.8.002.patch
        9 kB
        Yufei Gu
      3. YARN-6151.branch-2.8.003.patch
        11 kB
        Yufei Gu

        Issue Links

          Activity

          Hide
          yufeigu Yufei Gu added a comment - - edited

          Uploaded one simple solution which is before we decide which queue to preempt, we check if this queue can be preempted. For leaf queue, we check if the queue uses more resources than its fair share. For parent queue, we need to recursively go down to each level to check. There might be a performance concern, but considering the performance is still acceptable before YARN-5969 which did recursively get resource usage anyway, it should be fine.

          Show
          yufeigu Yufei Gu added a comment - - edited Uploaded one simple solution which is before we decide which queue to preempt, we check if this queue can be preempted. For leaf queue, we check if the queue uses more resources than its fair share. For parent queue, we need to recursively go down to each level to check. There might be a performance concern, but considering the performance is still acceptable before YARN-5969 which did recursively get resource usage anyway, it should be fine.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user flyrain opened a pull request:

          https://github.com/apache/hadoop/pull/188

          YARN-6151. FS Preemption doesn't filter out queues which cannot be pr…

          …eempted.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/flyrain/hadoop branch-2.8

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/188.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #188


          commit 5691fdff2b400571bb4669daf6e25791aaaa0db9
          Author: Yufei Gu <yufei.gu@cloudera.com>
          Date: 2017-02-06T20:57:05Z

          YARN-6151. FS Preemption doesn't filter out queues which cannot be preempted.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user flyrain opened a pull request: https://github.com/apache/hadoop/pull/188 YARN-6151 . FS Preemption doesn't filter out queues which cannot be pr… …eempted. You can merge this pull request into a Git repository by running: $ git pull https://github.com/flyrain/hadoop branch-2.8 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/188.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #188 commit 5691fdff2b400571bb4669daf6e25791aaaa0db9 Author: Yufei Gu <yufei.gu@cloudera.com> Date: 2017-02-06T20:57:05Z YARN-6151 . FS Preemption doesn't filter out queues which cannot be preempted.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 44s branch-2.8 passed
          +1 compile 0m 29s branch-2.8 passed with JDK v1.8.0_121
          +1 compile 0m 31s branch-2.8 passed with JDK v1.7.0_121
          +1 checkstyle 0m 18s branch-2.8 passed
          +1 mvnsite 0m 36s branch-2.8 passed
          +1 mvneclipse 0m 17s branch-2.8 passed
          +1 findbugs 1m 19s branch-2.8 passed
          +1 javadoc 0m 22s branch-2.8 passed with JDK v1.8.0_121
          +1 javadoc 0m 23s branch-2.8 passed with JDK v1.7.0_121
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 26s the patch passed with JDK v1.8.0_121
          +1 javac 0m 26s the patch passed
          +1 compile 0m 30s the patch passed with JDK v1.7.0_121
          +1 javac 0m 30s the patch passed
          +1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20)
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 17s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121
          -1 unit 76m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          170m 12s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Issue YARN-6151
          GITHUB PR https://github.com/apache/hadoop/pull/188
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cbdc539edb8e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 2bbcaa8
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14834/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14834/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 44s branch-2.8 passed +1 compile 0m 29s branch-2.8 passed with JDK v1.8.0_121 +1 compile 0m 31s branch-2.8 passed with JDK v1.7.0_121 +1 checkstyle 0m 18s branch-2.8 passed +1 mvnsite 0m 36s branch-2.8 passed +1 mvneclipse 0m 17s branch-2.8 passed +1 findbugs 1m 19s branch-2.8 passed +1 javadoc 0m 22s branch-2.8 passed with JDK v1.8.0_121 +1 javadoc 0m 23s branch-2.8 passed with JDK v1.7.0_121 +1 mvninstall 0m 30s the patch passed +1 compile 0m 26s the patch passed with JDK v1.8.0_121 +1 javac 0m 26s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_121 +1 javac 0m 30s the patch passed +1 checkstyle 0m 15s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121 -1 unit 76m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 170m 12s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue YARN-6151 GITHUB PR https://github.com/apache/hadoop/pull/188 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cbdc539edb8e 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 2bbcaa8 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14834/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14834/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99942869

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java —
          @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)
          }

          /**
          + * Recursively check if the queue can be preempted based on whether the
          + * resource usage is greater than fair share.
          + *
          + * @return true if the queue can be preempted
          + */
          + public boolean canBePreempted() {
          + assert parent != null;
          — End diff –

          Why is this necessary?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99942869 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java — @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold) } /** + * Recursively check if the queue can be preempted based on whether the + * resource usage is greater than fair share. + * + * @return true if the queue can be preempted + */ + public boolean canBePreempted() { + assert parent != null; — End diff – Why is this necessary?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99943676

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java —
          @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)
          }

          /**
          + * Recursively check if the queue can be preempted based on whether the
          + * resource usage is greater than fair share.
          + *
          + * @return true if the queue can be preempted
          + */
          + public boolean canBePreempted() {
          — End diff –

          Should the check of the allowPreemptionFrom flag also be part of this method?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99943676 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java — @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold) } /** + * Recursively check if the queue can be preempted based on whether the + * resource usage is greater than fair share. + * + * @return true if the queue can be preempted + */ + public boolean canBePreempted() { — End diff – Should the check of the allowPreemptionFrom flag also be part of this method?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99943541

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java —
          @@ -2036,10 +2036,10 @@ public void testPreemptionIsNotDelayedToNextRound() throws Exception

          { .getLeafQueue("queueA.queueA2", false), clock.getTime()); assertEquals(3277, toPreempt.getMemorySize()); - // verify if the 3 containers required by queueA2 are preempted in the same + // verify if the 4 containers required by queueA2 are preempted in the same // round scheduler.preemptResources(toPreempt); - assertEquals(3, scheduler.getSchedulerApp(app1).getPreemptionContainers() + assertEquals(4, scheduler.getSchedulerApp(app1).getPreemptionContainers() .size()); }

          — End diff –

          Can we add a new test that verifies the exact scenario in the JIRA description?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99943541 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java — @@ -2036,10 +2036,10 @@ public void testPreemptionIsNotDelayedToNextRound() throws Exception { .getLeafQueue("queueA.queueA2", false), clock.getTime()); assertEquals(3277, toPreempt.getMemorySize()); - // verify if the 3 containers required by queueA2 are preempted in the same + // verify if the 4 containers required by queueA2 are preempted in the same // round scheduler.preemptResources(toPreempt); - assertEquals(3, scheduler.getSchedulerApp(app1).getPreemptionContainers() + assertEquals(4, scheduler.getSchedulerApp(app1).getPreemptionContainers() .size()); } — End diff – Can we add a new test that verifies the exact scenario in the JIRA description?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user flyrain commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99944929

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java —
          @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)
          }

          /**
          + * Recursively check if the queue can be preempted based on whether the
          + * resource usage is greater than fair share.
          + *
          + * @return true if the queue can be preempted
          + */
          + public boolean canBePreempted() {
          — End diff –

          It should be, but allowPreemptionFrom is introduced after 2.8.x.

          Show
          githubbot ASF GitHub Bot added a comment - Github user flyrain commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99944929 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java — @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold) } /** + * Recursively check if the queue can be preempted based on whether the + * resource usage is greater than fair share. + * + * @return true if the queue can be preempted + */ + public boolean canBePreempted() { — End diff – It should be, but allowPreemptionFrom is introduced after 2.8.x.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Karthik Kambatla for the review. Uploaded patch 002 which adds a unit test having exactly same situation from the JIRA description.

          Show
          yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for the review. Uploaded patch 002 which adds a unit test having exactly same situation from the JIRA description.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99967724

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java —
          @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)
          }

          /**
          + * Recursively check if the queue can be preempted based on whether the
          + * resource usage is greater than fair share.
          + *
          + * @return true if the queue can be preempted
          + */
          + public boolean canBePreempted() {
          — End diff –

          Aah, I keep forgetting branch-2.8 was cut years ago.

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99967724 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java — @@ -236,6 +236,29 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold) } /** + * Recursively check if the queue can be preempted based on whether the + * resource usage is greater than fair share. + * + * @return true if the queue can be preempted + */ + public boolean canBePreempted() { — End diff – Aah, I keep forgetting branch-2.8 was cut years ago.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99968314

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java —
          @@ -242,7 +244,9 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold)

          • @return true if the queue can be preempted
            */
            public boolean canBePreempted() {
          • assert parent != null;
            + Preconditions.checkNotNull(parent, "Parent queue can't be null since"
              • End diff –

          Maybe, we could make this message more clear. "Parent queue is null. Looks like we are checking if root can be preempted."

          Alternatively, can we make the if check (parent != null && ...)? That way, else would capture the null case and things should work fine?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99968314 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java — @@ -242,7 +244,9 @@ public void setFairSharePreemptionThreshold(float fairSharePreemptionThreshold) @return true if the queue can be preempted */ public boolean canBePreempted() { assert parent != null; + Preconditions.checkNotNull(parent, "Parent queue can't be null since" End diff – Maybe, we could make this message more clear. "Parent queue is null. Looks like we are checking if root can be preempted." Alternatively, can we make the if check (parent != null && ...)? That way, else would capture the null case and things should work fine?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user kambatla commented on a diff in the pull request:

          https://github.com/apache/hadoop/pull/188#discussion_r99967958

          — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java —
          @@ -2043,6 +2043,78 @@ public void testPreemptionIsNotDelayedToNextRound() throws Exception

          { .size()); }

          + @Test
          + public void testPreemptionFilterOutNonPreemptableQueues() throws Exception {
          — End diff –

          Can we add this test to TestFairSchedulerPreemption instead?

          Show
          githubbot ASF GitHub Bot added a comment - Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/188#discussion_r99967958 — Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java — @@ -2043,6 +2043,78 @@ public void testPreemptionIsNotDelayedToNextRound() throws Exception { .size()); } + @Test + public void testPreemptionFilterOutNonPreemptableQueues() throws Exception { — End diff – Can we add this test to TestFairSchedulerPreemption instead?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 38s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 9m 26s branch-2.8 passed
          +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_121
          +1 compile 0m 35s branch-2.8 passed with JDK v1.7.0_121
          +1 checkstyle 0m 21s branch-2.8 passed
          +1 mvnsite 0m 43s branch-2.8 passed
          +1 mvneclipse 0m 18s branch-2.8 passed
          +1 findbugs 1m 24s branch-2.8 passed
          +1 javadoc 0m 28s branch-2.8 passed with JDK v1.8.0_121
          +1 javadoc 0m 26s branch-2.8 passed with JDK v1.7.0_121
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 29s the patch passed with JDK v1.8.0_121
          +1 javac 0m 29s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.7.0_121
          +1 javac 0m 31s the patch passed
          +1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20)
          +1 mvnsite 0m 35s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 17s the patch passed
          +1 javadoc 0m 18s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121
          -1 unit 77m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          175m 23s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestResourceTrackerService
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
          JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
            hadoop.yarn.server.resourcemanager.TestClientRMService
            hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
            hadoop.yarn.server.resourcemanager.TestAMAuthorization



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Issue YARN-6151
          GITHUB PR https://github.com/apache/hadoop/pull/188
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1e7f74159321 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 2bbcaa8
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14855/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14855/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14855/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 38s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 26s branch-2.8 passed +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_121 +1 compile 0m 35s branch-2.8 passed with JDK v1.7.0_121 +1 checkstyle 0m 21s branch-2.8 passed +1 mvnsite 0m 43s branch-2.8 passed +1 mvneclipse 0m 18s branch-2.8 passed +1 findbugs 1m 24s branch-2.8 passed +1 javadoc 0m 28s branch-2.8 passed with JDK v1.8.0_121 +1 javadoc 0m 26s branch-2.8 passed with JDK v1.7.0_121 +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_121 +1 javac 0m 29s the patch passed +1 compile 0m 31s the patch passed with JDK v1.7.0_121 +1 javac 0m 31s the patch passed +1 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 19 unchanged - 1 fixed = 19 total (was 20) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 17s the patch passed +1 javadoc 0m 18s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121 -1 unit 77m 6s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 175m 23s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.TestAMAuthorization JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestClientRMService   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue YARN-6151 GITHUB PR https://github.com/apache/hadoop/pull/188 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1e7f74159321 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 2bbcaa8 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14855/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14855/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14855/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 36s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          +1 mvninstall 11m 59s branch-2.8 passed
          +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_121
          +1 compile 0m 34s branch-2.8 passed with JDK v1.7.0_121
          +1 checkstyle 0m 20s branch-2.8 passed
          +1 mvnsite 0m 48s branch-2.8 passed
          +1 mvneclipse 0m 19s branch-2.8 passed
          +1 findbugs 1m 19s branch-2.8 passed
          +1 javadoc 0m 22s branch-2.8 passed with JDK v1.8.0_121
          +1 javadoc 0m 26s branch-2.8 passed with JDK v1.7.0_121
          +1 mvninstall 0m 37s the patch passed
          +1 compile 0m 31s the patch passed with JDK v1.8.0_121
          +1 javac 0m 31s the patch passed
          +1 compile 0m 34s the patch passed with JDK v1.7.0_121
          +1 javac 0m 34s the patch passed
          -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 20 unchanged - 1 fixed = 21 total (was 21)
          +1 mvnsite 0m 38s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 33s the patch passed
          +1 javadoc 0m 20s the patch passed with JDK v1.8.0_121
          +1 javadoc 0m 24s the patch passed with JDK v1.7.0_121
          -1 unit 77m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          192m 57s



          Reason Tests
          JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens
          JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
            hadoop.yarn.server.resourcemanager.TestAMAuthorization
            hadoop.yarn.server.resourcemanager.TestClientRMTokens



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:5af2af1
          JIRA Issue YARN-6151
          GITHUB PR https://github.com/apache/hadoop/pull/188
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 85759560d4f6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.8 / 2bbcaa8
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14857/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14857/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14857/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 36s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 11m 59s branch-2.8 passed +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_121 +1 compile 0m 34s branch-2.8 passed with JDK v1.7.0_121 +1 checkstyle 0m 20s branch-2.8 passed +1 mvnsite 0m 48s branch-2.8 passed +1 mvneclipse 0m 19s branch-2.8 passed +1 findbugs 1m 19s branch-2.8 passed +1 javadoc 0m 22s branch-2.8 passed with JDK v1.8.0_121 +1 javadoc 0m 26s branch-2.8 passed with JDK v1.7.0_121 +1 mvninstall 0m 37s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_121 +1 javac 0m 31s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_121 +1 javac 0m 34s the patch passed -0 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 20 unchanged - 1 fixed = 21 total (was 21) +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 24s the patch passed with JDK v1.7.0_121 -1 unit 77m 16s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 192m 57s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerLazyPreemption   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:5af2af1 JIRA Issue YARN-6151 GITHUB PR https://github.com/apache/hadoop/pull/188 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 85759560d4f6 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 2bbcaa8 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14857/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14857/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14857/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14857/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          +1. Just committed this to branch-2.8.

          Show
          kasha Karthik Kambatla added a comment - +1. Just committed this to branch-2.8.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for fixing this, Yufei.

          Show
          kasha Karthik Kambatla added a comment - Thanks for fixing this, Yufei.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Karthik Kambatla for the review and commit.

          Show
          yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for the review and commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user flyrain commented on the issue:

          https://github.com/apache/hadoop/pull/188

          Committed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user flyrain commented on the issue: https://github.com/apache/hadoop/pull/188 Committed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user flyrain closed the pull request at:

          https://github.com/apache/hadoop/pull/188

          Show
          githubbot ASF GitHub Bot added a comment - Github user flyrain closed the pull request at: https://github.com/apache/hadoop/pull/188

            People

            • Assignee:
              yufeigu Yufei Gu
              Reporter:
              yufeigu Yufei Gu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development