Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5890

FairScheduler should log information about AM-resource-usage and max-AM-share for queues

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      There are several cases where jobs in a queue or stuck likely because of maxAMShare. It is hard to debug these issues without any information.
      At the very least, we need to log both AM-resource-usage and max-AM-share for queues.

      1. YARN-5890.001.patch
        13 kB
        Yufei Gu
      2. YARN-5890.002.patch
        13 kB
        Yufei Gu
      3. YARN-5890.003.patch
        14 kB
        Yufei Gu
      4. YARN-5890.004.patch
        14 kB
        Yufei Gu
      5. YARN-5890.005.patch
        15 kB
        Yufei Gu
      6. YARN-5890.branch-2.001.patch
        15 kB
        Yufei Gu

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 53s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 19s trunk passed
        +1 findbugs 1m 1s trunk passed
        +1 javadoc 0m 26s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 0m 33s the patch passed
        +1 javac 0m 33s the patch passed
        -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 241 unchanged - 0 fixed = 247 total (was 241)
        +1 mvnsite 0m 39s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 6s the patch passed
        +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 934 unchanged - 1 fixed = 934 total (was 935)
        +1 unit 43m 16s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        59m 22s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5890
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839475/YARN-5890.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 8b5af6e550fc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / b4f1971
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13967/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13967/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13967/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 53s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 19s trunk passed +1 findbugs 1m 1s trunk passed +1 javadoc 0m 26s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 0m 33s the patch passed +1 javac 0m 33s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 241 unchanged - 0 fixed = 247 total (was 241) +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 6s the patch passed +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 934 unchanged - 1 fixed = 934 total (was 935) +1 unit 43m 16s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 59m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5890 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839475/YARN-5890.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8b5af6e550fc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b4f1971 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13967/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13967/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13967/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Thank you for the patch Yufei Gu!

        On the comment:

        If FairShare is zero, use min(maxShare, available resource) instead to prevent zero value for maximum AM resource since it forbids any job running in the queue.

        Just a note, you might also want to discuss the case, when fair share is not 0 but so low that we cannot launch any AM. This case might deserve a unit test as well, since the AM launch is forbidden, despite the fair share 0 case, when, it is allowed.

        set queueA and queueB weight zero.

        This is a typo. You need to mention queue1, queue2 and queue3. It would be helpful to give some more meaningful names to these queues like queueFSZeroWitMax, queueFSZero, queueFSOne.

        createSchedulingRequestExistingApplication(1024, 1, amPriority, attId1);
        ...
        assertEquals((long) ((memCapacity - 1024) * queue2.getMaxAMShare()),
        

        It would make sense to extract 1024 and 1 as variables and reuse the variable name here. It helps with readability.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the patch Yufei Gu ! On the comment: If FairShare is zero, use min(maxShare, available resource) instead to prevent zero value for maximum AM resource since it forbids any job running in the queue. Just a note, you might also want to discuss the case, when fair share is not 0 but so low that we cannot launch any AM. This case might deserve a unit test as well, since the AM launch is forbidden, despite the fair share 0 case, when, it is allowed. set queueA and queueB weight zero. This is a typo. You need to mention queue1, queue2 and queue3. It would be helpful to give some more meaningful names to these queues like queueFSZeroWitMax, queueFSZero, queueFSOne. createSchedulingRequestExistingApplication(1024, 1, amPriority, attId1); ... assertEquals(( long ) ((memCapacity - 1024) * queue2.getMaxAMShare()), It would make sense to extract 1024 and 1 as variables and reuse the variable name here. It helps with readability.
        Hide
        yufeigu Yufei Gu added a comment - - edited

        Thanks Miklos Szegedi for the review. I've uploaded the new patch for your comments.

        The case that fair share is not 0 but so low that we cannot launch any AM is kind of out of scope of the JIRA. We can definitely cover it later in other JIRA.
        For the clause memCapacity - 1024, 1024 refers to the AM resource used in the queue, so I use amResource.getMemorySize() to get it.

        Show
        yufeigu Yufei Gu added a comment - - edited Thanks Miklos Szegedi for the review. I've uploaded the new patch for your comments. The case that fair share is not 0 but so low that we cannot launch any AM is kind of out of scope of the JIRA. We can definitely cover it later in other JIRA. For the clause memCapacity - 1024 , 1024 refers to the AM resource used in the queue, so I use amResource.getMemorySize() to get it.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 50s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 59s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 242 unchanged - 0 fixed = 248 total (was 242)
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 934 unchanged - 1 fixed = 934 total (was 935)
        -1 unit 39m 5s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        54m 35s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5890
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839902/YARN-5890.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a1677adcc5be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 683e0c7
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14006/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14006/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 50s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 242 unchanged - 0 fixed = 248 total (was 242) +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 19s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 934 unchanged - 1 fixed = 934 total (was 935) -1 unit 39m 5s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 54m 35s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5890 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839902/YARN-5890.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a1677adcc5be 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 683e0c7 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14006/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14006/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14006/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14006/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        The unit test failure is unrelated.

        Show
        yufeigu Yufei Gu added a comment - The unit test failure is unrelated.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks, Yufei Gu. Minor points on the tests:

        • queueFSNoneZero should be queueFSNonZero.
        • Since max AM share isn't being set explicitly, it might be a good idea to add a comment stating what it you expect the default to be. It doesn't really affect anything, but as a reader of the code, it's the first question that comes to mind.
        • I'd love to see some messages on all those asserts.
        • I know that none of the methods here change amResource, but since Resource is mutable, I'd feel better if we created a new one for each part of the test rather than reusing the same amResource.
        Show
        templedf Daniel Templeton added a comment - Thanks, Yufei Gu . Minor points on the tests: queueFSNoneZero should be queueFSNonZero . Since max AM share isn't being set explicitly, it might be a good idea to add a comment stating what it you expect the default to be. It doesn't really affect anything, but as a reader of the code, it's the first question that comes to mind. I'd love to see some messages on all those asserts. I know that none of the methods here change amResource , but since Resource is mutable, I'd feel better if we created a new one for each part of the test rather than reusing the same amResource .
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Daniel Templeton. I uploaded the patch 003 for all your comments.
        I set the max AM share explicitly instead of add some doc, which I think more readable.

        Show
        yufeigu Yufei Gu added a comment - Thanks Daniel Templeton . I uploaded the patch 003 for all your comments. I set the max AM share explicitly instead of add some doc, which I think more readable.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 patch 0m 5s YARN-5890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



        Subsystem Report/Notes
        JIRA Issue YARN-5890
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840754/YARN-5890.003.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14088/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s YARN-5890 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-5890 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840754/YARN-5890.003.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/14088/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yufeigu Yufei Gu added a comment -

        Did the rebase in patch 004.

        Show
        yufeigu Yufei Gu added a comment - Did the rebase in patch 004.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 26s trunk passed
        +1 compile 0m 40s trunk passed
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 0m 46s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 12s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 40s the patch passed
        +1 compile 0m 37s the patch passed
        +1 javac 0m 37s the patch passed
        -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 238 unchanged - 0 fixed = 244 total (was 238)
        +1 mvnsite 0m 44s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 23s the patch passed
        +1 javadoc 0m 22s the patch passed
        +1 unit 39m 35s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        58m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5890
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840759/YARN-5890.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f45daf25e0ff 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / be88d57
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14090/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14090/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14090/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 26s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 46s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -0 checkstyle 0m 24s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 238 unchanged - 0 fixed = 244 total (was 238) +1 mvnsite 0m 44s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 39m 35s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 58m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5890 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12840759/YARN-5890.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f45daf25e0ff 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / be88d57 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14090/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14090/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14090/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the update, Yufei Gu. Looks good. I like the assert messages. After taking a closer look at the patch, I have one outstanding concern: can you explain this math to me? Math.abs(maxAMShare - -1.0f Why subtract -1 instead of adding 1? Why subtract -1 at all?

        Show
        templedf Daniel Templeton added a comment - Thanks for the update, Yufei Gu . Looks good. I like the assert messages. After taking a closer look at the patch, I have one outstanding concern: can you explain this math to me? Math.abs(maxAMShare - -1.0f Why subtract -1 instead of adding 1? Why subtract -1 at all?
        Hide
        templedf Daniel Templeton added a comment -

        Ah, I see now. It was Robert Kanter who introduced that code. You're just moving it around.

        Show
        templedf Daniel Templeton added a comment - Ah, I see now. It was Robert Kanter who introduced that code. You're just moving it around.
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Daniel Templeton. -1.0f is the value to disable the maxAMShare introduced in YARN-2187.

        Show
        yufeigu Yufei Gu added a comment - Thanks Daniel Templeton . -1.0f is the value to disable the maxAMShare introduced in YARN-2187 .
        Hide
        templedf Daniel Templeton added a comment -

        One more thing. Does FSQueue.getMaxAMShare() need to be public?

        Also, it would be really swell to add javadocs for the new methods in FSQueueMetrics. Yes, they would be the only ones with javadocs, but hopefully you'll start a new trend.

        Show
        templedf Daniel Templeton added a comment - One more thing. Does FSQueue.getMaxAMShare() need to be public? Also, it would be really swell to add javadocs for the new methods in FSQueueMetrics . Yes, they would be the only ones with javadocs, but hopefully you'll start a new trend.
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Daniel Templeton's review. I uploaded the patch 005 for your comments.

        Show
        yufeigu Yufei Gu added a comment - Thanks Daniel Templeton 's review. I uploaded the patch 005 for your comments.
        Hide
        templedf Daniel Templeton added a comment -

        LGTM +1

        Show
        templedf Daniel Templeton added a comment - LGTM +1
        Hide
        templedf Daniel Templeton added a comment -

        Committed to trunk. Can you please supply a branch-2 patch?

        Show
        templedf Daniel Templeton added a comment - Committed to trunk. Can you please supply a branch-2 patch?
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10908/)
        YARN-5890. FairScheduler should log information about AM-resource-usage (templedf: rev 3b9d3acd203cef4d861c5182fc4dccc55128d347)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10908 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10908/ ) YARN-5890 . FairScheduler should log information about AM-resource-usage (templedf: rev 3b9d3acd203cef4d861c5182fc4dccc55128d347) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueMetrics.java
        Hide
        yufeigu Yufei Gu added a comment -

        Thanks Daniel Templeton. I uploaded the branch-2 patch.

        Show
        yufeigu Yufei Gu added a comment - Thanks Daniel Templeton . I uploaded the branch-2 patch.
        Hide
        templedf Daniel Templeton added a comment -

        Thanks for the fix, Yufei Gu. Committed to trunk and branch-2.

        Show
        templedf Daniel Templeton added a comment - Thanks for the fix, Yufei Gu . Committed to trunk and branch-2.
        Hide
        djp Junping Du added a comment -

        This patch goes to branch-2 only instead of 2.8, set 2.9 as fix version.

        Show
        djp Junping Du added a comment - This patch goes to branch-2 only instead of 2.8, set 2.9 as fix version.

          People

          • Assignee:
            yufeigu Yufei Gu
            Reporter:
            yufeigu Yufei Gu
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development