Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5899

Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: capacity scheduler
    • Labels:
      None

      Description

      A small fix inside function canAssignToThisQueue() for printing DEBUG info. Please see patch attached.

      1. YARN-5899.002.patch
        4 kB
        Ying Zhang
      2. YARN-5899.001.patch
        2 kB
        Ying Zhang

        Activity

        Hide
        sunilg Sunil G added a comment -

        Could you please attach a patch here.

        Show
        sunilg Sunil G added a comment - Could you please attach a patch here.
        Hide
        Ying Zhang Ying Zhang added a comment -

        Hi Sunil G, patch attached. Thank you.

        Show
        Ying Zhang Ying Zhang added a comment - Hi Sunil G , patch attached. Thank you.
        Hide
        Ying Zhang Ying Zhang added a comment -

        It is just a small fix. I believe the intention is to print some debug information when we are able to assign to the queue, i.e., returning true. (See YARN-3361 where this change was made at the first place.)

        Show
        Ying Zhang Ying Zhang added a comment - It is just a small fix. I believe the intention is to print some debug information when we are able to assign to the queue, i.e., returning true. (See YARN-3361 where this change was made at the first place.)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 58s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 1s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 5s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 41m 45s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        57m 18s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5899
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839753/YARN-5899.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2e80844a03a9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c65d6b6
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13990/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13990/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 58s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 1s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 5s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 41m 45s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 57m 18s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5899 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839753/YARN-5899.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2e80844a03a9 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c65d6b6 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13990/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13990/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13990/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Ying Zhang Ying Zhang added a comment -

        The UT failure is known and tracked by YARN-5548.

        Show
        Ying Zhang Ying Zhang added a comment - The UT failure is known and tracked by YARN-5548 .
        Hide
        sunilg Sunil G added a comment -

        Thanks for the patch. I ll take a look today.

        Show
        sunilg Sunil G added a comment - Thanks for the patch. I ll take a look today.
        Hide
        sunilg Sunil G added a comment - - edited

        Hi Ying Zhang

        Your observation is correct. We are using that debug log to get some more information when assign to queue is passed. So your patch is correct. I think we can also add one more debug info in case when we return false case also (wherever we have current debug log). We could print

        LOG.debug("Failed to assignToQueue: " + getQueueName() + " usedResources: " + queueUsage.getUsed() + ", clusterResources: " + clusterResource +  ", maxLimitCapacity: " + currentLimitResource + ", currTotalUsed:" + usedExceptKillable);		
        

        Thoughts?

        Show
        sunilg Sunil G added a comment - - edited Hi Ying Zhang Your observation is correct. We are using that debug log to get some more information when assign to queue is passed. So your patch is correct. I think we can also add one more debug info in case when we return false case also (wherever we have current debug log). We could print LOG.debug("Failed to assignToQueue: " + getQueueName() + " usedResources: " + queueUsage.getUsed() + ", clusterResources: " + clusterResource + ", maxLimitCapacity: " + currentLimitResource + ", currTotalUsed:" + usedExceptKillable); Thoughts?
        Hide
        Ying Zhang Ying Zhang added a comment - - edited

        Hi Sunil G, thanks for the review. Agree. I've updated the patch, please take a look. Besides those you've listed here, I'm thinking "resourceCouldBeUnreserved" would also be useful.

        Show
        Ying Zhang Ying Zhang added a comment - - edited Hi Sunil G , thanks for the review. Agree. I've updated the patch, please take a look. Besides those you've listed here, I'm thinking "resourceCouldBeUnreserved" would also be useful.
        Hide
        sunilg Sunil G added a comment -

        Thanks. Looks fine for me. Will wait for jenkins to come back.

        Show
        sunilg Sunil G added a comment - Thanks. Looks fine for me. Will wait for jenkins to come back.
        Hide
        Ying Zhang Ying Zhang added a comment - - edited

        Not sure why there wasn't Jenkins result. Delete and re-attach YARN-5899.002.patch to try to start Jenkins again.

        Show
        Ying Zhang Ying Zhang added a comment - - edited Not sure why there wasn't Jenkins result. Delete and re-attach YARN-5899 .002.patch to try to start Jenkins again.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 25s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 3s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 7s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 40m 44s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        62m 36s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5899
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845468/YARN-5899.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b2ab3f90ec0b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8fadd69
        Default Java 1.8.0_111
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14545/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14545/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 25s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 7s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 40m 44s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 62m 36s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5899 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845468/YARN-5899.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2ab3f90ec0b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8fadd69 Default Java 1.8.0_111 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14545/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14545/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Ying Zhang Ying Zhang added a comment - - edited

        This patch only introduces some modification to debug information. No test case is needed. I've enabled DEBUG mode and test it manually. Failed test case TestRMRestart.testFinishedAppRemovalAfterRMRestart is known and tracked by YARN-5548.

        Show
        Ying Zhang Ying Zhang added a comment - - edited This patch only introduces some modification to debug information. No test case is needed. I've enabled DEBUG mode and test it manually. Failed test case TestRMRestart.testFinishedAppRemovalAfterRMRestart is known and tracked by YARN-5548 .
        Hide
        sunilg Sunil G added a comment -

        +1. Committing later today if there are no objections.

        Show
        sunilg Sunil G added a comment - +1. Committing later today if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        Thanks Ying Zhang for the contribution. Committed to trunk and branch-2.

        Show
        sunilg Sunil G added a comment - Thanks Ying Zhang for the contribution. Committed to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11097/)
        YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs (sunilg: rev 9594c35dcb655add1991d8fd15897b40c4ad6205)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #11097 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11097/ ) YARN-5899 . Debug log in AbstractCSQueue#canAssignToThisQueue needs (sunilg: rev 9594c35dcb655add1991d8fd15897b40c4ad6205) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
        Hide
        Ying Zhang Ying Zhang added a comment -

        Thanks Sunil G for the review and commit.

        Show
        Ying Zhang Ying Zhang added a comment - Thanks Sunil G for the review and commit.

          People

          • Assignee:
            Ying Zhang Ying Zhang
            Reporter:
            Ying Zhang Ying Zhang
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development