Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5731

Preemption calculation is not accurate when reserved containers are present in queue.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      YARN Capacity Scheduler does not kick Preemption under below scenario.
      Two queues A and B each with 50% capacity and 100% maximum capacity and user limit factor 2. Minimum Container size is 1536MB and total cluster resource is 40GB. Now submit the first job which needs 1536MB for AM and 9 task containers each 4.5GB to queue A. Job will get 8 containers total (AM 1536MB + 7 * 4.5GB = 33GB) and the cluster usage is 93.8% and the job has reserved a container of 4.5GB.
      Now when next job (1536MB for AM and 9 task containers each 4.5GB) is submitted onto queue B. The job hangs in ACCEPTED state forever and RM scheduler never kicks in Preemption. (RM UI Image 2 attached)
      Test Case:

      ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue A --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 1000000

      After a minute..

      ./spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --queue B --executor-memory 4G --executor-cores 4 --num-executors 9 ../lib/spark-examples*.jar 1000000

      Credit to: Prabhu Joseph for bug investigation and troubleshooting.

      1. YARN-5731.001.patch
        12 kB
        Wangda Tan
      2. YARN-5731.002.patch
        10 kB
        Wangda Tan
      3. YARN-5731.addendum.003.patch
        6 kB
        Wangda Tan
      4. YARN-5731.addendum.004.patch
        7 kB
        Wangda Tan
      5. YARN-5731.branch-2.002.patch
        10 kB
        Wangda Tan
      6. YARN-5731.branch-2.8.004.patch
        12 kB
        Wangda Tan
      7. YARN-5731-branch-2.8.001.patch
        34 kB
        Sunil G
      8. YARN-5731-branch-2.8.001.patch
        6 kB
        Wangda Tan

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached patch for the fix.

        Show
        leftnoteasy Wangda Tan added a comment - Attached patch for the fix.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.8 Compile Tests
        +1 mvninstall 9m 53s branch-2.8 passed
        +1 compile 0m 35s branch-2.8 passed with JDK v1.8.0_131
        +1 compile 0m 36s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 20s branch-2.8 passed
        +1 mvnsite 0m 44s branch-2.8 passed
        +1 findbugs 1m 24s branch-2.8 passed
        +1 javadoc 0m 26s branch-2.8 passed with JDK v1.8.0_131
        +1 javadoc 0m 28s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 39s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.8.0_131
        +1 javac 0m 30s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_131
        +1 javac 0m 34s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 27s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 76m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        175m 12s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicyForReservedContainers
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicyForReservedContainers
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876493/YARN-5731-branch-2.8.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 46c6dc67f3d9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 0f9f7bb
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16356/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16356/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.8 Compile Tests +1 mvninstall 9m 53s branch-2.8 passed +1 compile 0m 35s branch-2.8 passed with JDK v1.8.0_131 +1 compile 0m 36s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 20s branch-2.8 passed +1 mvnsite 0m 44s branch-2.8 passed +1 findbugs 1m 24s branch-2.8 passed +1 javadoc 0m 26s branch-2.8 passed with JDK v1.8.0_131 +1 javadoc 0m 28s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 39s the patch passed +1 compile 0m 30s the patch passed with JDK v1.8.0_131 +1 javac 0m 30s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_131 +1 javac 0m 34s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 27s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 76m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 175m 12s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicyForReservedContainers   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicyForReservedContainers   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSurgicalPreemption   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876493/YARN-5731-branch-2.8.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 46c6dc67f3d9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 0f9f7bb Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/16356/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16356/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16356/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Test case failures seems related.
        I think since we enable this flag, we might miss one container to preempt. May be we need to recheck these test cases, thoughts?

        Show
        sunilg Sunil G added a comment - Test case failures seems related. I think since we enable this flag, we might miss one container to preempt. May be we need to recheck these test cases, thoughts?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached patch for trunk, I think we should fix this corner case in trunk as well.

        cc: Sunil G.

        Show
        leftnoteasy Wangda Tan added a comment - Attached patch for trunk, I think we should fix this corner case in trunk as well. cc: Sunil G .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 7s YARN-5731 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876922/YARN-5731.001.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16391/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s YARN-5731 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876922/YARN-5731.001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/16391/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Rebased to proper trunk version.

        Show
        leftnoteasy Wangda Tan added a comment - Rebased to proper trunk version.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 36s trunk passed
        +1 compile 0m 40s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 findbugs 1m 14s trunk passed
        +1 javadoc 0m 23s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 37s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 48 unchanged - 1 fixed = 57 total (was 49)
        +1 mvnsite 0m 35s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 8s the patch passed
        +1 javadoc 0m 19s the patch passed
              Other Tests
        -1 unit 42m 50s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        65m 16s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876937/YARN-5731.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 73fb7100f2de 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b628d0d
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16395/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16395/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16395/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16395/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 36s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 40s trunk passed +1 findbugs 1m 14s trunk passed +1 javadoc 0m 23s trunk passed       Patch Compile Tests +1 mvninstall 0m 37s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 48 unchanged - 1 fixed = 57 total (was 49) +1 mvnsite 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 8s the patch passed +1 javadoc 0m 19s the patch passed       Other Tests -1 unit 42m 50s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 65m 16s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876937/YARN-5731.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 73fb7100f2de 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b628d0d Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16395/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16395/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16395/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16395/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        +1 on latest patch. Committing the same if there are no objections.

        Show
        sunilg Sunil G added a comment - +1 on latest patch. Committing the same if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk. Thanks Wangda Tan. I guess we need a new patch for branch-2.8

        Show
        sunilg Sunil G added a comment - Committed to trunk. Thanks Wangda Tan . I guess we need a new patch for branch-2.8
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11999 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11999/)
        YARN-5731. Preemption calculation is not accurate when reserved (sunilg: rev cf0d0844d6ae25d537391edb9b65fca05d1848e6)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/FifoCandidatesSelector.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerPreemptionTestBase.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerSurgicalPreemption.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11999 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11999/ ) YARN-5731 . Preemption calculation is not accurate when reserved (sunilg: rev cf0d0844d6ae25d537391edb9b65fca05d1848e6) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/FifoCandidatesSelector.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerPreemptionTestBase.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerSurgicalPreemption.java
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached branch-2 patch.

        Show
        leftnoteasy Wangda Tan added a comment - Attached branch-2 patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2 Compile Tests
        +1 mvninstall 7m 10s branch-2 passed
        +1 compile 0m 33s branch-2 passed with JDK v1.8.0_131
        +1 compile 0m 35s branch-2 passed with JDK v1.7.0_131
        +1 checkstyle 0m 25s branch-2 passed
        +1 mvnsite 0m 38s branch-2 passed
        +1 findbugs 1m 18s branch-2 passed
        +1 javadoc 0m 26s branch-2 passed with JDK v1.8.0_131
        +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.8.0_131
        +1 javac 0m 33s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_131
        +1 javac 0m 34s the patch passed
        -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 48 unchanged - 1 fixed = 57 total (was 49)
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 28s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 58m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        120m 19s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
        JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877153/YARN-5731.branch-2.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8b373af568c4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / d83e871
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16428/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16428/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16428/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16428/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2 Compile Tests +1 mvninstall 7m 10s branch-2 passed +1 compile 0m 33s branch-2 passed with JDK v1.8.0_131 +1 compile 0m 35s branch-2 passed with JDK v1.7.0_131 +1 checkstyle 0m 25s branch-2 passed +1 mvnsite 0m 38s branch-2 passed +1 findbugs 1m 18s branch-2 passed +1 javadoc 0m 26s branch-2 passed with JDK v1.8.0_131 +1 javadoc 0m 25s branch-2 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 34s the patch passed +1 compile 0m 33s the patch passed with JDK v1.8.0_131 +1 javac 0m 33s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_131 +1 javac 0m 34s the patch passed -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 48 unchanged - 1 fixed = 57 total (was 49) +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 28s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 58m 58s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 120m 19s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler JDK v1.7.0_131 Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877153/YARN-5731.branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8b373af568c4 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d83e871 Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16428/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16428/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16428/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16428/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk and branch-2. Thank you very much Wangda Tan

        Show
        sunilg Sunil G added a comment - Committed to trunk and branch-2. Thank you very much Wangda Tan
        Hide
        leftnoteasy Wangda Tan added a comment -

        Instead of making the logics to be hard coded, we discussed and decided to make the behavior configurable.

        See javadocs of:
        CapacitySchedulerConfiguration#ADDTIONAL_BALANCE_QUEUE_BASED_ON_RESERVED_RESOURCE for more details.

        Show
        leftnoteasy Wangda Tan added a comment - Instead of making the logics to be hard coded, we discussed and decided to make the behavior configurable. See javadocs of: CapacitySchedulerConfiguration#ADDTIONAL_BALANCE_QUEUE_BASED_ON_RESERVED_RESOURCE for more details.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Offline suggested by Sunil G, uploaded patch with updated config names.

        Show
        leftnoteasy Wangda Tan added a comment - Offline suggested by Sunil G , uploaded patch with updated config names.
        Hide
        sunilg Sunil G added a comment -

        +1 on latest addendum patch. Committing tomorrow if there are no objections.

        Show
        sunilg Sunil G added a comment - +1 on latest addendum patch. Committing tomorrow if there are no objections.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 32s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 findbugs 1m 0s trunk passed
        +1 javadoc 0m 21s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 83 unchanged - 7 fixed = 92 total (was 90)
        +1 mvnsite 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 18s the patch passed
              Other Tests
        -1 unit 42m 32s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        64m 9s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877630/YARN-5731.addendum.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c3dbf74b20a1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0e78ae
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16466/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16466/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16466/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16466/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 32s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 36s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 21s trunk passed       Patch Compile Tests +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 9 new + 83 unchanged - 7 fixed = 92 total (was 90) +1 mvnsite 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests -1 unit 42m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 64m 9s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877630/YARN-5731.addendum.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c3dbf74b20a1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0e78ae Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16466/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16466/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16466/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16466/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 51s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 findbugs 1m 0s trunk passed
        +1 javadoc 0m 22s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        -0 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 83 unchanged - 7 fixed = 89 total (was 90)
        +1 mvnsite 0m 37s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 7s the patch passed
        +1 javadoc 0m 18s the patch passed
              Other Tests
        -1 unit 42m 39s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        64m 47s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877632/YARN-5731.addendum.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ecab00194f10 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b0e78ae
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16467/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16467/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16467/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 51s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 36s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 22s trunk passed       Patch Compile Tests +1 mvninstall 0m 33s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed -0 checkstyle 0m 26s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 83 unchanged - 7 fixed = 89 total (was 90) +1 mvnsite 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 7s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests -1 unit 42m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 64m 47s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877632/YARN-5731.addendum.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ecab00194f10 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b0e78ae Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16467/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16467/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16467/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16467/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Committed addendum patch. Thanks Wangda Tan

        Show
        sunilg Sunil G added a comment - Committed addendum patch. Thanks Wangda Tan
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12024/)
        Addendum patch for YARN-5731 (sunilg: rev 0b7afc060c2024a882bd1934d0f722bfca731742)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerSurgicalPreemption.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12024 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12024/ ) Addendum patch for YARN-5731 (sunilg: rev 0b7afc060c2024a882bd1934d0f722bfca731742) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerSurgicalPreemption.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java
        Hide
        sunilg Sunil G added a comment -

        Attaching branch-2.8 patch

        Show
        sunilg Sunil G added a comment - Attaching branch-2.8 patch
        Hide
        sunilg Sunil G added a comment -

        Reopening to run jenkins for branch-2.8 patch

        Show
        sunilg Sunil G added a comment - Reopening to run jenkins for branch-2.8 patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 25s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2.8 Compile Tests
        +1 mvninstall 8m 7s branch-2.8 passed
        +1 compile 0m 42s branch-2.8 passed with JDK v1.8.0_131
        +1 compile 0m 35s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 21s branch-2.8 passed
        +1 mvnsite 0m 41s branch-2.8 passed
        +1 findbugs 1m 16s branch-2.8 passed
        +1 javadoc 0m 21s branch-2.8 passed with JDK v1.8.0_131
        +1 javadoc 0m 23s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        -1 mvninstall 0m 28s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 compile 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_131.
        -1 javac 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_131.
        -1 compile 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        -1 javac 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 67 unchanged - 0 fixed = 72 total (was 67)
        -1 mvnsite 0m 30s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 21s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 21s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        18m 46s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879097/YARN-5731-branch-2.8.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e450a77c262b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / c586b7a
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16569/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16569/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2.8 Compile Tests +1 mvninstall 8m 7s branch-2.8 passed +1 compile 0m 42s branch-2.8 passed with JDK v1.8.0_131 +1 compile 0m 35s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 21s branch-2.8 passed +1 mvnsite 0m 41s branch-2.8 passed +1 findbugs 1m 16s branch-2.8 passed +1 javadoc 0m 21s branch-2.8 passed with JDK v1.8.0_131 +1 javadoc 0m 23s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests -1 mvninstall 0m 28s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_131. -1 javac 0m 26s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.8.0_131. -1 compile 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. -1 javac 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 67 unchanged - 0 fixed = 72 total (was 67) -1 mvnsite 0m 30s hadoop-yarn-server-resourcemanager in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 18m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879097/YARN-5731-branch-2.8.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e450a77c262b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / c586b7a Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_131.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16569/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16569/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16569/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached patch to branch-2.8, since branch-2.8 doesn't include the main fix, so I consolidated main fix and addendum fix to the same patch. (004)

        Show
        leftnoteasy Wangda Tan added a comment - Attached patch to branch-2.8, since branch-2.8 doesn't include the main fix, so I consolidated main fix and addendum fix to the same patch. (004)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 8m 27s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2.8 Compile Tests
        +1 mvninstall 6m 52s branch-2.8 passed
        +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_131
        +1 compile 0m 33s branch-2.8 passed with JDK v1.7.0_131
        +1 checkstyle 0m 20s branch-2.8 passed
        +1 mvnsite 0m 38s branch-2.8 passed
        +1 findbugs 1m 13s branch-2.8 passed
        +1 javadoc 0m 20s branch-2.8 passed with JDK v1.8.0_131
        +1 javadoc 0m 25s branch-2.8 passed with JDK v1.7.0_131
              Patch Compile Tests
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 24s the patch passed with JDK v1.8.0_131
        +1 javac 0m 24s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.7.0_131
        +1 javac 0m 29s the patch passed
        -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 68 unchanged - 0 fixed = 73 total (was 68)
        +1 mvnsite 0m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 21s the patch passed
        +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131
              Other Tests
        -1 unit 76m 8s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        176m 33s



        Reason Tests
        JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d946387
        JIRA Issue YARN-5731
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879734/YARN-5731.branch-2.8.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux de4c6f0c1503 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 0020a8e
        Default Java 1.7.0_131
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16631/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16631/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt
        JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16631/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16631/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 8m 27s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2.8 Compile Tests +1 mvninstall 6m 52s branch-2.8 passed +1 compile 0m 31s branch-2.8 passed with JDK v1.8.0_131 +1 compile 0m 33s branch-2.8 passed with JDK v1.7.0_131 +1 checkstyle 0m 20s branch-2.8 passed +1 mvnsite 0m 38s branch-2.8 passed +1 findbugs 1m 13s branch-2.8 passed +1 javadoc 0m 20s branch-2.8 passed with JDK v1.8.0_131 +1 javadoc 0m 25s branch-2.8 passed with JDK v1.7.0_131       Patch Compile Tests +1 mvninstall 0m 31s the patch passed +1 compile 0m 24s the patch passed with JDK v1.8.0_131 +1 javac 0m 24s the patch passed +1 compile 0m 29s the patch passed with JDK v1.7.0_131 +1 javac 0m 29s the patch passed -0 checkstyle 0m 16s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 68 unchanged - 0 fixed = 73 total (was 68) +1 mvnsite 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 17s the patch passed with JDK v1.8.0_131 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_131       Other Tests -1 unit 76m 8s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_131. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 176m 33s Reason Tests JDK v1.8.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_131 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:d946387 JIRA Issue YARN-5731 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879734/YARN-5731.branch-2.8.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux de4c6f0c1503 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 0020a8e Default Java 1.7.0_131 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16631/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16631/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_131.txt JDK v1.7.0_131 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16631/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16631/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to branch-2.8 as well.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to branch-2.8 as well.
        Hide
        djp Junping Du added a comment -

        Sounds like we forget to commit to branch-2.8.2. Just commit it.

        Show
        djp Junping Du added a comment - Sounds like we forget to commit to branch-2.8.2. Just commit it.

          People

          • Assignee:
            leftnoteasy Wangda Tan
            Reporter:
            sunilg Sunil G
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development