Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6769

Make schedulables without demand less needy in FairSharePolicy#compare

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.9.0, 3.0.0-beta1
    • Component/s: fairscheduler
    • Labels:
      None
    • Target Version/s:

      Description

      When use fairsheduler as RM scheduler, before assign container we will sort all queues or applications.
      We will use FairSharePolicy#compare as the comparator´╝î but the comparator is not so perfect.
      It have a problem as blow:
      1. when a queue use resource over minShare(minResources), it will put behind the queue whose demand is zeor.
      so it will greater opportunity to get the resource although it do not want. It will waste schedule time when assign container
      to queue or application.

      I have fix it, and I will upload the patch to the jira.

      1. YARN-6769.001.patch
        2 kB
        YunFan Zhou
      2. YARN-6769.002.patch
        7 kB
        YunFan Zhou
      3. YARN-6769.003.patch
        5 kB
        YunFan Zhou
      4. YARN-6769.004.patch
        5 kB
        YunFan Zhou

        Activity

        Hide
        daemon YunFan Zhou added a comment -
        diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        index f8cdb45929..e930b80e45 100644
        --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        @@ -79,6 +79,19 @@ public String getName() {
         
             @Override
             public int compare(Schedulable s1, Schedulable s2) {
        +      Resource demand1 = s1.getDemand();
        +      Resource demand2 = s2.getDemand();
        +      // Put the schedulable which does not require resource to
        +      // the end. So the other schedulable can get resource as soon as
        +      // possible though it use resource greater then it minShare or demand.
        +      if (demand1.equals(Resources.none()) &&
        +          !demand2.equals(Resources.none())) {
        +        return 1;
        +      } else if (demand2.equals(Resources.none()) &&
        +          !demand1.equals(Resources.none())) {
        +        return -1;
        +      }
        +      
               double minShareRatio1, minShareRatio2;
               double useToWeightRatio1, useToWeightRatio2;
               double weight1, weight2;
        @@ -86,9 +99,9 @@ public int compare(Schedulable s1, Schedulable s2) {
               Resource resourceUsage1 = s1.getResourceUsage();
               Resource resourceUsage2 = s2.getResourceUsage();
               Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null,
        -          s1.getMinShare(), s1.getDemand());
        +          s1.getMinShare(), demand1);
               Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null,
        -          s2.getMinShare(), s2.getDemand());
        +          s2.getMinShare(), demand2);
               boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
                   resourceUsage1, minShare1);
               boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null,
        
        
        Show
        daemon YunFan Zhou added a comment - diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java index f8cdb45929..e930b80e45 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java @@ -79,6 +79,19 @@ public String getName() { @Override public int compare(Schedulable s1, Schedulable s2) { + Resource demand1 = s1.getDemand(); + Resource demand2 = s2.getDemand(); + // Put the schedulable which does not require resource to + // the end. So the other schedulable can get resource as soon as + // possible though it use resource greater then it minShare or demand. + if (demand1.equals(Resources.none()) && + !demand2.equals(Resources.none())) { + return 1; + } else if (demand2.equals(Resources.none()) && + !demand1.equals(Resources.none())) { + return -1; + } + double minShareRatio1, minShareRatio2; double useToWeightRatio1, useToWeightRatio2; double weight1, weight2; @@ -86,9 +99,9 @@ public int compare(Schedulable s1, Schedulable s2) { Resource resourceUsage1 = s1.getResourceUsage(); Resource resourceUsage2 = s2.getResourceUsage(); Resource minShare1 = Resources.min(RESOURCE_CALCULATOR, null , - s1.getMinShare(), s1.getDemand()); + s1.getMinShare(), demand1); Resource minShare2 = Resources.min(RESOURCE_CALCULATOR, null , - s2.getMinShare(), s2.getDemand()); + s2.getMinShare(), demand2); boolean s1Needy = Resources.lessThan(RESOURCE_CALCULATOR, null , resourceUsage1, minShare1); boolean s2Needy = Resources.lessThan(RESOURCE_CALCULATOR, null ,
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks YunFan Zhou for working on this. Daniel Templeton has added you as a contributor. I assign this jira to you, and you can do this by yourself in any other jiras. Could you please upload a patch file? Thanks.

        Show
        yufeigu Yufei Gu added a comment - Thanks YunFan Zhou for working on this. Daniel Templeton has added you as a contributor. I assign this jira to you, and you can do this by yourself in any other jiras. Could you please upload a patch file? Thanks.
        Hide
        daemon YunFan Zhou added a comment -

        yufei Thanks yufei. I have already upload my patch file, what is the next I should do?

        Show
        daemon YunFan Zhou added a comment - yufei Thanks yufei. I have already upload my patch file, what is the next I should do?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
              trunk Compile Tests
        +1 mvninstall 13m 59s trunk passed
        +1 compile 0m 38s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 22s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 0m 37s the patch passed
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        +1 findbugs 1m 6s the patch passed
        +1 javadoc 0m 19s the patch passed
              Other Tests
        -1 unit 43m 6s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        65m 43s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6769
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876291/YARN-6769.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fc1c085f0a27 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f484a6f
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16340/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16340/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16340/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16340/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.       trunk Compile Tests +1 mvninstall 13m 59s trunk passed +1 compile 0m 38s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 39s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 22s trunk passed       Patch Compile Tests +1 mvninstall 0m 34s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 37s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 1m 6s the patch passed +1 javadoc 0m 19s the patch passed       Other Tests -1 unit 43m 6s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 65m 43s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6769 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876291/YARN-6769.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fc1c085f0a27 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f484a6f Default Java 1.8.0_131 findbugs v3.1.0-RC1 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16340/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16340/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16340/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16340/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daemon YunFan Zhou added a comment -

        Hadoop QA I am sorry for my break the test, so I fix it and upload a new patch file.
        Yufei Gu yufei, please help me review the code if you have free time, thanks a lot.

        Show
        daemon YunFan Zhou added a comment - Hadoop QA I am sorry for my break the test, so I fix it and upload a new patch file. Yufei Gu yufei, please help me review the code if you have free time, thanks a lot.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 1s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 51s trunk passed
        +1 compile 0m 35s trunk passed
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 findbugs 1m 3s trunk passed
        +1 javadoc 0m 21s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 33s the patch passed
        +1 javac 0m 33s the patch passed
        -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11)
        +1 mvnsite 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 21s the patch passed
              Other Tests
        -1 unit 46m 43s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        68m 57s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6769
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876322/YARN-6769.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3196b55a23cf 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 626d730
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16341/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16341/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16341/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 1s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 51s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 37s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 21s trunk passed       Patch Compile Tests +1 mvninstall 0m 33s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 11 unchanged - 0 fixed = 13 total (was 11) +1 mvnsite 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 21s the patch passed       Other Tests -1 unit 46m 43s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 68m 57s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6769 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876322/YARN-6769.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3196b55a23cf 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 626d730 Default Java 1.8.0_131 findbugs v3.1.0-RC1 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16341/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16341/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16341/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16341/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daemon YunFan Zhou added a comment -

        Daniel Templeton hi, Daniel. Can you help review my code? Pretty thanks!

        Show
        daemon YunFan Zhou added a comment - Daniel Templeton hi, Daniel. Can you help review my code? Pretty thanks!
        Hide
        yufeigu Yufei Gu added a comment -

        The patch looks good to me generally. Some thoughts:

        1. The logic in FairSharePolicy#compare looks good. However, the comment is verbose. Can you rephrase to just clarify that schedulable with no demand is less needy, and update the method Java doc as well?
        2. Why we need to change the recursive condition to genSchedulable.size() == 4 in method generateAndTest?
        3. Could you fix the checkstyle issue found by Hadoop QA
        Show
        yufeigu Yufei Gu added a comment - The patch looks good to me generally. Some thoughts: The logic in FairSharePolicy#compare looks good. However, the comment is verbose. Can you rephrase to just clarify that schedulable with no demand is less needy, and update the method Java doc as well? Why we need to change the recursive condition to genSchedulable.size() == 4 in method generateAndTest ? Could you fix the checkstyle issue found by Hadoop QA
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 12m 49s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 findbugs 1m 0s trunk passed
        +1 javadoc 0m 19s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 17s the patch passed
              Other Tests
        -1 unit 43m 36s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        64m 12s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6769
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876562/YARN-6769.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bb8358b0af65 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / fce7951
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16364/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16364/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16364/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 12m 49s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 35s trunk passed +1 findbugs 1m 0s trunk passed +1 javadoc 0m 19s trunk passed       Patch Compile Tests +1 mvninstall 0m 31s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 17s the patch passed       Other Tests -1 unit 43m 36s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 64m 12s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6769 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12876562/YARN-6769.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bb8358b0af65 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fce7951 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16364/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16364/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16364/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daemon YunFan Zhou added a comment -

        Yufei Gu Thanks yufei. You are right, I have already fix those problems.

        Show
        daemon YunFan Zhou added a comment - Yufei Gu Thanks yufei. You are right, I have already fix those problems.
        Hide
        yufeigu Yufei Gu added a comment - - edited

        Thanks for the new patch. Some nits:

        1. Can we change Java doc to "Schedulables without resource demand get lower priority than ones who have demands."? Better to avoid using Resources.none since descriptive words are better than code in Java doc.
        2. Is it better to use Resources.greaterThan(RESOURCE_CALCULATOR, null, demand2, Resources.none()) instead of !demand2.equals(Resources.none())? Resource could be negative. Maybe it's not possible for inputs of this method, but the former is less error-prone anyway.
        Show
        yufeigu Yufei Gu added a comment - - edited Thanks for the new patch. Some nits: Can we change Java doc to "Schedulables without resource demand get lower priority than ones who have demands."? Better to avoid using Resources.none since descriptive words are better than code in Java doc. Is it better to use Resources.greaterThan(RESOURCE_CALCULATOR, null, demand2, Resources.none()) instead of !demand2.equals(Resources.none()) ? Resource could be negative. Maybe it's not possible for inputs of this method, but the former is less error-prone anyway.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 13m 3s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 findbugs 0m 58s trunk passed
        +1 javadoc 0m 21s trunk passed
              Patch Compile Tests
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 18s the patch passed
              Other Tests
        -1 unit 42m 25s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        63m 32s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6769
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877033/YARN-6769.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 50e8e7358c26 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e15e271
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16412/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16412/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests +1 mvninstall 13m 3s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 35s trunk passed +1 findbugs 0m 58s trunk passed +1 javadoc 0m 21s trunk passed       Patch Compile Tests +1 mvninstall 0m 32s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 33s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 18s the patch passed       Other Tests -1 unit 42m 25s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 63m 32s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6769 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12877033/YARN-6769.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 50e8e7358c26 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e15e271 Default Java 1.8.0_131 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/16412/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16412/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/16412/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        daemon YunFan Zhou added a comment -

        Yufei Gu hi, yufei. Is there any other problems in my new patch?

        Show
        daemon YunFan Zhou added a comment - Yufei Gu hi, yufei. Is there any other problems in my new patch?
        Hide
        yufeigu Yufei Gu added a comment -

        LGTM. +1.

        Show
        yufeigu Yufei Gu added a comment - LGTM. +1.
        Hide
        yufeigu Yufei Gu added a comment -

        Hi YunFan Zhou, we usually put contributor's name in git logs, something like "YARN-6769. xxxx (Daemon via Yufei Gu)". I assume daemon is not your really name. Do you want to use your really name in the git log? If Yes, please send me your name. Thanks.

        Show
        yufeigu Yufei Gu added a comment - Hi YunFan Zhou , we usually put contributor's name in git logs, something like " YARN-6769 . xxxx (Daemon via Yufei Gu)". I assume daemon is not your really name. Do you want to use your really name in the git log? If Yes, please send me your name. Thanks.
        Hide
        daemon YunFan Zhou added a comment -

        Yufei Gu, Thanks yufei. I really name is zhouyunfan.
        Thank you so mush for doing so mush for me!

        Show
        daemon YunFan Zhou added a comment - Yufei Gu , Thanks yufei. I really name is zhouyunfan. Thank you so mush for doing so mush for me!
        Hide
        yufeigu Yufei Gu added a comment -

        Committed to trunk and branch-2. Thanks YunFan Zhou for working on this. Congratulations for your first Hadoop commit!

        Show
        yufeigu Yufei Gu added a comment - Committed to trunk and branch-2. Thanks YunFan Zhou for working on this. Congratulations for your first Hadoop commit!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12006 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12006/)
        YARN-6769. Make schedulables without demand less needy in (yufei: rev 4a574e9a84f2e997038452b22f2ad2a2d42e8ac8)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingPolicy.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12006 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12006/ ) YARN-6769 . Make schedulables without demand less needy in (yufei: rev 4a574e9a84f2e997038452b22f2ad2a2d42e8ac8) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingPolicy.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java

          People

          • Assignee:
            daemon YunFan Zhou
            Reporter:
            daemon YunFan Zhou
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development