Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5077

Fix FSLeafQueue#getFairShare() for queues with zero fairshare

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns <memory:0, vCores:0>
      2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns <memory:16384, vCores:8>
      In case 1), that means no container ever gets allocated for an AM because from the viewpoint of the RM, there is never any headroom to allocate a container on that queue.

      For example, we have a pool with the following weights:

      • root.dev 0.0
      • root.product 1.0

      The root.dev is a best effort pool and should only get resources if root.product is not running. In our tests, with no jobs running under root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and never start.

      1. YARN-5077.001.patch
        12 kB
        Yufei Gu
      2. YARN-5077.002.patch
        12 kB
        Yufei Gu
      3. YARN-5077.003.patch
        12 kB
        Yufei Gu
      4. YARN-5077.004.patch
        14 kB
        Yufei Gu
      5. YARN-5077.005.patch
        14 kB
        Yufei Gu
      6. YARN-5077.006.patch
        6 kB
        Yufei Gu
      7. YARN-5077.007.patch
        6 kB
        Yufei Gu
      8. YARN-5077.008.patch
        15 kB
        Yufei Gu
      9. YARN-5077.009.patch
        15 kB
        Yufei Gu
      10. YARN-5077.010.patch
        15 kB
        Yufei Gu

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 15s trunk passed
        +1 compile 0m 35s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 53s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        -1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 5 new + 213 unchanged - 0 fixed = 218 total (was 213)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 26s the patch passed
        -1 unit 30m 27s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s Patch does not generate ASF License warnings.
        44m 34s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804471/YARN-5077.001.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c842db42e1b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34fddd1
        Default Java 1.8.0_91
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11504/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11504/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 15s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed -1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: patch generated 5 new + 213 unchanged - 0 fixed = 218 total (was 213) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 26s the patch passed -1 unit 30m 27s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s Patch does not generate ASF License warnings. 44m 34s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804471/YARN-5077.001.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c842db42e1b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34fddd1 Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11504/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11504/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11504/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        The patch 002 solves the style issues. All failed tests are unrelated.

        Show
        yufeigu Yufei Gu added a comment - The patch 002 solves the style issues. All failed tests are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 2s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 53s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 8s the patch passed
        +1 javadoc 0m 25s the patch passed
        -1 unit 30m 12s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s Patch does not generate ASF License warnings.
        43m 49s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804487/YARN-5077.002.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c8360e017da3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34fddd1
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11508/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11508/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11508/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11508/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 2s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 53s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 8s the patch passed +1 javadoc 0m 25s the patch passed -1 unit 30m 12s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s Patch does not generate ASF License warnings. 43m 49s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804487/YARN-5077.002.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c8360e017da3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34fddd1 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11508/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11508/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11508/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11508/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        All failed tests are unrelated.

        Show
        yufeigu Yufei Gu added a comment - All failed tests are unrelated.
        Hide
        kasha Karthik Kambatla added a comment -

        This has been a long standing inconvenience. Thanks for working on this.

        High-level comment: IIUC, we want to consider queues with zero weight only when computing instantaneous fairshare. And, IIRR, only active apps are passed to compute-instantaneous-shares. So, we probably don't have to check if a queue is active. That said when computing instantaneous fairshares, we could check if any of the queues have a non-zero weight.

        Other minor comments on the patch:

        1. Instead of double negation in the variable name, can we pass forceWeightToOne to ComputeShares#computeShare and allWeightsZero to resourceUsedWithWeightToResourceRatio?
        2. The method to check weights itself could be areAllWeightsZero

        The test is pretty neat. I cringe every time I see the xml form of the FairScheduler allocations file in tests, but we already have many of them. Filed YARN-5016 for that.

        Show
        kasha Karthik Kambatla added a comment - This has been a long standing inconvenience. Thanks for working on this. High-level comment: IIUC, we want to consider queues with zero weight only when computing instantaneous fairshare. And, IIRR, only active apps are passed to compute-instantaneous-shares. So, we probably don't have to check if a queue is active. That said when computing instantaneous fairshares, we could check if any of the queues have a non-zero weight. Other minor comments on the patch: Instead of double negation in the variable name, can we pass forceWeightToOne to ComputeShares#computeShare and allWeightsZero to resourceUsedWithWeightToResourceRatio ? The method to check weights itself could be areAllWeightsZero The test is pretty neat. I cringe every time I see the xml form of the FairScheduler allocations file in tests, but we already have many of them. Filed YARN-5016 for that.
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Karthik Kambatla for reviewing.

        To filter out the non-active queue is done by handleFixedFairShares, which is invoked by computeSharesInternal. But handleFixedFairShares does more than that, it also filters out zero-weight queue, and that's one of place we want to fix in this JIRA. If there are all zero-weight active queues, we should get them in instead of removing them. So I should know if there is no non-zero-weight active queue before handleFixedFairShares, which is done by func noNonZeroWeightActive.

        I totally agree we should avoid double negation name, but it seems reasonable here. Of course, the name could be called allWeightsZero after function handleFixedFairShares, so I modify the name after that as your suggestions.

        I assume you said YARN-5106. It is fair enough.

        Show
        yufeigu Yufei Gu added a comment - Thanks Karthik Kambatla for reviewing. To filter out the non-active queue is done by handleFixedFairShares , which is invoked by computeSharesInternal . But handleFixedFairShares does more than that, it also filters out zero-weight queue, and that's one of place we want to fix in this JIRA. If there are all zero-weight active queues, we should get them in instead of removing them. So I should know if there is no non-zero-weight active queue before handleFixedFairShares , which is done by func noNonZeroWeightActive . I totally agree we should avoid double negation name, but it seems reasonable here. Of course, the name could be called allWeightsZero after function handleFixedFairShares , so I modify the name after that as your suggestions. I assume you said YARN-5106 . It is fair enough.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 10s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 0s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 29m 36s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        43m 2s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804740/YARN-5077.003.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4ede70affbdb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 42c22f7
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11581/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11581/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11581/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11581/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 10s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 29m 36s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 43m 2s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804740/YARN-5077.003.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4ede70affbdb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 42c22f7 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11581/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11581/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11581/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11581/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        In that case, should we update the way handleFixedShares addresses queues with zero weight? May be, that method should include zero-weight queues in nonFixedSchedulables that it constructs? That way, the helper method being added in this patch doesn't have to recheck if the queue is active?

        For the tests themselves, should we add a test to see that if a set minshare for a queue with zero weight, it actually gets the minshare allocated?

        Show
        kasha Karthik Kambatla added a comment - In that case, should we update the way handleFixedShares addresses queues with zero weight? May be, that method should include zero-weight queues in nonFixedSchedulables that it constructs? That way, the helper method being added in this patch doesn't have to recheck if the queue is active? For the tests themselves, should we add a test to see that if a set minshare for a queue with zero weight, it actually gets the minshare allocated?
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, thanks a lot for the reviewing. I agree with you completely and uploaded a new patch for it.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , thanks a lot for the reviewing. I agree with you completely and uploaded a new patch for it.
        Hide
        yufeigu Yufei Gu added a comment -

        BTW, we already have a test case testFairShareWithZeroWeightNoneZeroMinRes to test zero weight queue with a minResource. I modified it a little to let it make more sense.

        Show
        yufeigu Yufei Gu added a comment - BTW, we already have a test case testFairShareWithZeroWeightNoneZeroMinRes to test zero weight queue with a minResource. I modified it a little to let it make more sense.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 6m 19s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 1s the patch passed
        +1 javadoc 0m 20s the patch passed
        -1 unit 34m 39s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        48m 15s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestRMAdminService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806021/YARN-5077.004.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c6fecd6ba385 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / edd716e
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11679/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11679/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 6m 19s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 34m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 48m 15s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806021/YARN-5077.004.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c6fecd6ba385 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / edd716e Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11679/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11679/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11679/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Actually, one last nit. Sorry for not suggesting this earlier.

        In ComputeFairShares#computeShare, if forceWeightOne is set, I don't see the need to check the weights. Also, it can be simplified with a ternary operator. And, we can drop the comments as well.

        float weight = forceWeightOne ? 1 : sched.getWeights().getWeight(type);
        
        Show
        kasha Karthik Kambatla added a comment - Actually, one last nit. Sorry for not suggesting this earlier. In ComputeFairShares#computeShare, if forceWeightOne is set, I don't see the need to check the weights. Also, it can be simplified with a ternary operator. And, we can drop the comments as well. float weight = forceWeightOne ? 1 : sched.getWeights().getWeight(type);
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, thanks for the detailed review. I uploaded patch 005 with the modification you suggested.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , thanks for the detailed review. I uploaded patch 005 with the modification you suggested.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 5m 59s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 51s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 26s the patch passed
        +1 javac 0m 26s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 37s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 0m 21s the patch passed
        -1 unit 34m 51s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s Patch does not generate ASF License warnings.
        48m 23s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806183/YARN-5077.005.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 47c5d9efe0d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 77d5ce9
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11692/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11692/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 5m 59s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 26s the patch passed +1 javac 0m 26s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 21s the patch passed -1 unit 34m 51s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s Patch does not generate ASF License warnings. 48m 23s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806183/YARN-5077.005.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 47c5d9efe0d8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 77d5ce9 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11692/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11692/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11692/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        And, another thing came to mind. Sorry about this.

        The current patch works when none of the non-zero-weight queues are active. However, if one of the non-zero-weight queues is active but is using only a fraction of the cluster (say, 10%), none of the zero-weight queues would have resources allocated.

        We could potentially get this in and then look into improving it to address the other case as well. Or, if it is simple enough, we could address it here.

        Show
        kasha Karthik Kambatla added a comment - And, another thing came to mind. Sorry about this. The current patch works when none of the non-zero-weight queues are active. However, if one of the non-zero-weight queues is active but is using only a fraction of the cluster (say, 10%), none of the zero-weight queues would have resources allocated. We could potentially get this in and then look into improving it to address the other case as well. Or, if it is simple enough, we could address it here.
        Hide
        kasha Karthik Kambatla added a comment -

        Actually, thinking more about this, I am beginning to doubt our initial approach. Or, it has just been a long day.

        A queue should get resources assigned even if the fairshare is zero. However, the AM cannot be spawned because the maxAMShare is zero if the fairshare is zero. Yufei Gu - can you confirm if this is the case?

        If this is indeed the case, we should probably look at this maxAMShare thing more comprehensively.

        Show
        kasha Karthik Kambatla added a comment - Actually, thinking more about this, I am beginning to doubt our initial approach. Or, it has just been a long day. A queue should get resources assigned even if the fairshare is zero. However, the AM cannot be spawned because the maxAMShare is zero if the fairshare is zero. Yufei Gu - can you confirm if this is the case? If this is indeed the case, we should probably look at this maxAMShare thing more comprehensively.
        Hide
        yufeigu Yufei Gu added a comment - - edited

        Karthik Kambatla, I have this concern as well. If a queue can get resources assigned even if the fairshare is zero, why we have this bug. I first tested it with a unmanaged AM. It can get the resources even its weight is zero. Then a managed AM cannot get the resources because its maxAMResource always is zero, and maxAMResource is calculated by multiplying faireshare and maxAMShare.

        Do you suggest we can change the algorithm of calculating maxAMResource to solve this problem?

        Show
        yufeigu Yufei Gu added a comment - - edited Karthik Kambatla , I have this concern as well. If a queue can get resources assigned even if the fairshare is zero, why we have this bug. I first tested it with a unmanaged AM. It can get the resources even its weight is zero. Then a managed AM cannot get the resources because its maxAMResource always is zero, and maxAMResource is calculated by multiplying faireshare and maxAMShare. Do you suggest we can change the algorithm of calculating maxAMResource to solve this problem?
        Hide
        kasha Karthik Kambatla added a comment -

        Updating the way we calculate maxAMResource seems like a promising way of addressing this. Since we have had other issues with this (the most recent one fixed as part of YARN-4866), it would be nice to take a more comprehensive look at this.

        Show
        kasha Karthik Kambatla added a comment - Updating the way we calculate maxAMResource seems like a promising way of addressing this. Since we have had other issues with this (the most recent one fixed as part of YARN-4866 ), it would be nice to take a more comprehensive look at this.
        Hide
        yufeigu Yufei Gu added a comment - - edited

        Uploaded patch 006 to update the way we calculate maxAMShare, so that a zero-weight AM can get resources no matter whether there are active non-zero weight queues.

        Show
        yufeigu Yufei Gu added a comment - - edited Uploaded patch 006 to update the way we calculate maxAMShare, so that a zero-weight AM can get resources no matter whether there are active non-zero weight queues.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 19s trunk passed
        +1 compile 0m 28s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 32s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 51s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 25s the patch passed
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 19s the patch passed
        +1 mvnsite 0m 29s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 0m 54s the patch passed
        +1 javadoc 0m 20s the patch passed
        -1 unit 35m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s Patch does not generate ASF License warnings.
        48m 38s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
          hadoop.yarn.server.resourcemanager.TestAMAuthorization



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806762/YARN-5077.006.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 13daf65b038b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 21890c4
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11751/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11751/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11751/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11751/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 19s trunk passed +1 compile 0m 28s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 32s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 51s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 27s the patch passed +1 compile 0m 25s the patch passed +1 javac 0m 25s the patch passed +1 checkstyle 0m 19s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 54s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 35m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s Patch does not generate ASF License warnings. 48m 38s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens   hadoop.yarn.server.resourcemanager.TestAMAuthorization Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806762/YARN-5077.006.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 13daf65b038b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 21890c4 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11751/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11751/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11751/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11751/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        The maxShare corresponds to maxResources. It is not necessary that every queue configures maxResources. When not configured, this resolves to minResources which could be 0.

        Show
        kasha Karthik Kambatla added a comment - The maxShare corresponds to maxResources. It is not necessary that every queue configures maxResources. When not configured, this resolves to minResources which could be 0.
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, thanks for the review.

        IIUC, if a queue is not configured maxResources, maxShare will be the Integer.MAX_VALUE. Am I missing something?

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , thanks for the review. IIUC, if a queue is not configured maxResources, maxShare will be the Integer.MAX_VALUE. Am I missing something?
        Hide
        kasha Karthik Kambatla added a comment -

        My bad again. I should probably take some time off. maxShare will be Integer.MAX_VALUE, but that is also an issue. What happens if the cluster resources are smaller than maxShare? Wouldn't we run into the same livelock issue maxAMShare was meant to solve?

        Given the number of issues surrounding this code, I wonder if there is a fundamental issue here that needs a more comprehensive look.

        Show
        kasha Karthik Kambatla added a comment - My bad again. I should probably take some time off. maxShare will be Integer.MAX_VALUE, but that is also an issue. What happens if the cluster resources are smaller than maxShare? Wouldn't we run into the same livelock issue maxAMShare was meant to solve? Given the number of issues surrounding this code, I wonder if there is a fundamental issue here that needs a more comprehensive look.
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, you are right. It might be a livelock here. We can use all the available resources of the cluster instead of use maxShare to calculate the maxAMResource. I uploaded patch 007 for it.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , you are right. It might be a livelock here. We can use all the available resources of the cluster instead of use maxShare to calculate the maxAMResource. I uploaded patch 007 for it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 26s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 4s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 33m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        48m 41s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
          hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
          hadoop.yarn.server.resourcemanager.TestRMAdminService



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:2c91fd8
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807826/YARN-5077.007.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux bfd77adda509 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 97e2449
        Default Java 1.8.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11828/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/11828/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 26s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 33m 23s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 48m 41s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestClientRMTokens   hadoop.yarn.server.resourcemanager.TestRMAdminService Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12807826/YARN-5077.007.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux bfd77adda509 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 97e2449 Default Java 1.8.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/11828/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/11828/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/11828/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Interesting approach on the last patch.

        Few comments:

        1. Can we extend it to address YARN-4866 as well, so we have a uniform approach?
        2. Instead of checking for weight, we might want to check if fairshare memory/cpu being 0. That way, we will also address cases where the weight is really small due to which the fairshare is essentially 0.
        3. FSQueue#getMaxShare appears to be not checking the parent queues. Shouldn't we be checking that? FWIW, I am not a fan of our current approach of querying AllocationConfiguration. Will it be better to use FSQueue to store queue-specific information instead? I am comfortable with tackling that in another JIRA either before or immediately after this.
        Show
        kasha Karthik Kambatla added a comment - Interesting approach on the last patch. Few comments: Can we extend it to address YARN-4866 as well, so we have a uniform approach? Instead of checking for weight, we might want to check if fairshare memory/cpu being 0. That way, we will also address cases where the weight is really small due to which the fairshare is essentially 0. FSQueue#getMaxShare appears to be not checking the parent queues. Shouldn't we be checking that? FWIW, I am not a fan of our current approach of querying AllocationConfiguration. Will it be better to use FSQueue to store queue-specific information instead? I am comfortable with tackling that in another JIRA either before or immediately after this.
        Hide
        yufeigu Yufei Gu added a comment - - edited

        Karthik Kambatla, Thanks for the review.

        I will discuss with you offline with 1 and 2. I will create a new JIRA for 3. YARN-5264 is the JIRA for the 3.

        Show
        yufeigu Yufei Gu added a comment - - edited Karthik Kambatla , Thanks for the review. I will discuss with you offline with 1 and 2. I will create a new JIRA for 3. YARN-5264 is the JIRA for the 3.
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, thanks you very much for your comments. I uploaded patch 008 for all comments.

        The new patch consolidates YARN-4866, add a test case with tiny weight besides the zero weight, and remove useless functions.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , thanks you very much for your comments. I uploaded patch 008 for all comments. The new patch consolidates YARN-4866 , add a test case with tiny weight besides the zero weight, and remove useless functions.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 4s YARN-5077 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811228/YARN-5077.008.patch
        JIRA Issue YARN-5077
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12050/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s YARN-5077 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811228/YARN-5077.008.patch JIRA Issue YARN-5077 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12050/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        Upload patch 009 to fix the patch cannot apply issue.

        Show
        yufeigu Yufei Gu added a comment - Upload patch 009 to fix the patch cannot apply issue.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 31s trunk passed
        +1 compile 0m 29s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 33s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 54s trunk passed
        +1 javadoc 0m 20s trunk passed
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 27s the patch passed
        -1 javac 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
        +1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 253 unchanged - 4 fixed = 253 total (was 257)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 57s the patch passed
        +1 javadoc 0m 16s the patch passed
        +1 unit 31m 21s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        45m 1s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811248/YARN-5077.009.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dfd0ff03488a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bf78040
        Default Java 1.8.0_91
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/12056/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12056/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12056/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 31s trunk passed +1 compile 0m 29s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 54s trunk passed +1 javadoc 0m 20s trunk passed +1 mvninstall 0m 28s the patch passed +1 compile 0m 27s the patch passed -1 javac 0m 27s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) +1 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 253 unchanged - 4 fixed = 253 total (was 257) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 57s the patch passed +1 javadoc 0m 16s the patch passed +1 unit 31m 21s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 45m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811248/YARN-5077.009.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dfd0ff03488a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf78040 Default Java 1.8.0_91 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12056/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12056/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12056/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        The javac warning appears to be due to the changes introduced by YARN-4844. Can you update the call to getMemory to use getMemorySize instead?

        Show
        kasha Karthik Kambatla added a comment - The javac warning appears to be due to the changes introduced by YARN-4844 . Can you update the call to getMemory to use getMemorySize instead?
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, uploaded patch 010 for it.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , uploaded patch 010 for it.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 10s trunk passed
        +1 compile 0m 30s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 0m 55s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 29s the patch passed
        +1 compile 0m 29s the patch passed
        +1 javac 0m 29s the patch passed
        +1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 253 unchanged - 4 fixed = 253 total (was 257)
        +1 mvnsite 0m 35s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 0s the patch passed
        +1 javadoc 0m 20s the patch passed
        +1 unit 35m 19s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        51m 14s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e2f6409
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811404/YARN-5077.010.patch
        JIRA Issue YARN-5077
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f4d932eb687c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2800695
        Default Java 1.8.0_91
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12066/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12066/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 10s trunk passed +1 compile 0m 30s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 0m 55s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 29s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 23s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 253 unchanged - 4 fixed = 253 total (was 257) +1 mvnsite 0m 35s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 0s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 35m 19s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 51m 14s Subsystem Report/Notes Docker Image:yetus/hadoop:e2f6409 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12811404/YARN-5077.010.patch JIRA Issue YARN-5077 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f4d932eb687c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2800695 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12066/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12066/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        +1. Checking this in.

        Show
        kasha Karthik Kambatla added a comment - +1. Checking this in.
        Hide
        kasha Karthik Kambatla added a comment -

        Just committed to trunk and branch-2.

        Thanks Yufei for the contribution, especially for persisting through the several iterations of reviews and the need to re-pivot. The latest patch is quite nicely done.

        Excited to see this in; this fixes a long-standing usability issue for FairScheduler users.

        Show
        kasha Karthik Kambatla added a comment - Just committed to trunk and branch-2. Thanks Yufei for the contribution, especially for persisting through the several iterations of reviews and the need to re-pivot. The latest patch is quite nicely done. Excited to see this in; this fixes a long-standing usability issue for FairScheduler users.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #9981 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9981/)
        YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero (kasha: rev 20f2799938c5cad59f59617cdb0f86e137c3c2c0)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9981 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9981/ ) YARN-5077 . Fix FSLeafQueue#getFairShare() for queues with zero (kasha: rev 20f2799938c5cad59f59617cdb0f86e137c3c2c0) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FifoPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java
        Hide
        yufeigu Yufei Gu added a comment -

        Karthik Kambatla, thank you very much for detailed review, nice suggestions and the final check it.

        Show
        yufeigu Yufei Gu added a comment - Karthik Kambatla , thank you very much for detailed review, nice suggestions and the final check it.

          People

          • Assignee:
            yufeigu Yufei Gu
            Reporter:
            yufeigu Yufei Gu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development