Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3489

RMServerUtils.validateResourceRequests should only obtain queue info once

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Since the label support was added we now get the queue info for each request being validated in SchedulerUtils.validateResourceRequest. If validateResourceRequests needs to validate a lot of requests at a time (e.g.: large cluster with lots of varied locality in the requests) then it will get the queue info for each request. Since we build the queue info this generates a lot of unnecessary garbage, as the queue isn't changing between requests. We should grab the queue info once and pass it down rather than building it again for each request.

      1. YARN-3489.01.patch
        6 kB
        Varun Saxena
      2. YARN-3489.02.patch
        3 kB
        Varun Saxena
      3. YARN-3489.03.patch
        5 kB
        Varun Saxena
      4. YARN-3489-branch-2.7.patch
        5 kB
        Wangda Tan
      5. YARN-3489-branch-2.7.02.patch
        4 kB
        Varun Saxena
      6. YARN-3489-branch-2.7.03.patch
        4 kB
        Varun Saxena
      7. YARN-3489-branch-2.7.04.patch
        5 kB
        Varun Saxena

        Activity

        Hide
        varun_saxena Varun Saxena added a comment -

        Thanks Wangda Tan for the review and commit

        Show
        varun_saxena Varun Saxena added a comment - Thanks Wangda Tan for the review and commit
        Hide
        leftnoteasy Wangda Tan added a comment -

        I tried to run tests of RM locally after applied the patch against branch-2.7, all tests can get passed. Committed to branch-2.7, thanks Varun Saxena!

        Show
        leftnoteasy Wangda Tan added a comment - I tried to run tests of RM locally after applied the patch against branch-2.7, all tests can get passed. Committed to branch-2.7, thanks Varun Saxena !
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12735791/YARN-3489-branch-2.7.04.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 5450413
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8111/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735791/YARN-3489-branch-2.7.04.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 5450413 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8111/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Wangda Tan, rebased the patch. Cant we kick jenkins for branch-2.7 ?

        Show
        varun_saxena Varun Saxena added a comment - Wangda Tan , rebased the patch. Cant we kick jenkins for branch-2.7 ?
        Hide
        varun_saxena Varun Saxena added a comment -

        Wangda Tan, sorry missed your comment...Will have a look.

        Show
        varun_saxena Varun Saxena added a comment - Wangda Tan , sorry missed your comment...Will have a look.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Varun Saxena,
        Tx for updating, I tried to run RM tests locally after applied the patch, a couple of test failures:

        Results :

        Failed tests:
        TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche:383->testRMWritingMassiveHistory:441 null

        Tests in error:
        TestAppManager.testRMAppSubmitDuplicateApplicationId:531 » NullPointer
        TestAppManager.testRMAppSubmitMaxAppAttempts:506 » NullPointer
        TestAppManager.testRMAppSubmit:463 » NullPointer
        TestClientRMService.testAppSubmit:859 » NullPointer
        TestClientRMService.testGetApplications:959 » NullPointer
        TestClientRMService.testConcurrentAppSubmit:1115 » test timed out after 4000 ...

        Could you look at them?

        Show
        leftnoteasy Wangda Tan added a comment - Varun Saxena , Tx for updating, I tried to run RM tests locally after applied the patch, a couple of test failures: Results : Failed tests: TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche:383->testRMWritingMassiveHistory:441 null Tests in error: TestAppManager.testRMAppSubmitDuplicateApplicationId:531 » NullPointer TestAppManager.testRMAppSubmitMaxAppAttempts:506 » NullPointer TestAppManager.testRMAppSubmit:463 » NullPointer TestClientRMService.testAppSubmit:859 » NullPointer TestClientRMService.testGetApplications:959 » NullPointer TestClientRMService.testConcurrentAppSubmit:1115 » test timed out after 4000 ... Could you look at them?
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12733319/YARN-3489-branch-2.7.03.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / b0ad644
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7960/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733319/YARN-3489-branch-2.7.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b0ad644 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7960/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Varun Saxena, patch doesn't apply to latest branch-2.7, could you please check?

        Show
        leftnoteasy Wangda Tan added a comment - Varun Saxena , patch doesn't apply to latest branch-2.7, could you please check?
        Hide
        varun_saxena Varun Saxena added a comment -

        Wangda Tan, updated branch-2.7 patch. Ran RM tests and failures were unrelated to code change

        Show
        varun_saxena Varun Saxena added a comment - Wangda Tan , updated branch-2.7 patch. Ran RM tests and failures were unrelated to code change
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732569/YARN-3489-branch-2.7.02.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 065d8f2
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7919/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732569/YARN-3489-branch-2.7.02.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 065d8f2 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7919/console This message was automatically generated.
        Hide
        varun_saxena Varun Saxena added a comment -

        Ok...Will do so

        Show
        varun_saxena Varun Saxena added a comment - Ok...Will do so
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #193 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/193/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #183 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/183/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2123 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2123/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2141 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2141/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #194 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/194/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #925 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/925/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java hadoop-yarn-project/CHANGES.txt
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        -1 patch 0m 0s The patch command could not apply the patch during dryrun.



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12732137/YARN-3489-branch-2.7.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision branch-2 / 12584ac
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7877/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732137/YARN-3489-branch-2.7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision branch-2 / 12584ac Console output https://builds.apache.org/job/PreCommit-YARN-Build/7877/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Varun Saxena,
        I noticed there're some test failures of the patch I rebased, could you take a look at the patch I attached and run tests of yarn-server-resourcemanager? It seems like there're some test environment setup issues.

        Show
        leftnoteasy Wangda Tan added a comment - Varun Saxena , I noticed there're some test failures of the patch I rebased, could you take a look at the patch I attached and run tests of yarn-server-resourcemanager? It seems like there're some test environment setup issues.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk/branch-2, branch-2.7 needs rebase, running tests locally. Will commit once all RM related tests can get passed.

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk/branch-2, branch-2.7 needs rebase, running tests locally. Will commit once all RM related tests can get passed.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7800 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7800/)
        YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7800 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7800/ ) YARN-3489 . RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda) (wangda: rev d6f6741296639a73f5306e3ebefec84a40ca03e5) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committing.

        Show
        leftnoteasy Wangda Tan added a comment - Committing.
        Hide
        varun_saxena Varun Saxena added a comment -

        Wangda Tan, test case passes in local. Failure seems unrelated.

        Show
        varun_saxena Varun Saxena added a comment - Wangda Tan , test case passes in local. Failure seems unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 14m 41s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 32s There were no new javac warning messages.
        +1 javadoc 9m 41s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 0m 52s The applied patch generated 1 new checkstyle issues (total was 76, now 77).
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 install 1m 38s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 findbugs 1m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        -1 yarn tests 52m 6s Tests failed in hadoop-yarn-server-resourcemanager.
            88m 45s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12731613/YARN-3489.03.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / ed0f4db
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7833/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
        hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7833/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7833/testReport/
        Java 1.7.0_55
        uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7833/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 41s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 0m 52s The applied patch generated 1 new checkstyle issues (total was 76, now 77). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 16s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 yarn tests 52m 6s Tests failed in hadoop-yarn-server-resourcemanager.     88m 45s   Reason Tests Failed unit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12731613/YARN-3489.03.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ed0f4db checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7833/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7833/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7833/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7833/console This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Patch LGTM, will commit once Jenkins get back.

        Show
        leftnoteasy Wangda Tan added a comment - Patch LGTM, will commit once Jenkins get back.
        Hide
        varun_saxena Varun Saxena added a comment -

        Wangda Tan, oh didn't notice that SchedulerUtils is marked as private. We can change the current API then because my major concern was changing a public API. I will update new patch as per your suggestion then.

        Show
        varun_saxena Varun Saxena added a comment - Wangda Tan , oh didn't notice that SchedulerUtils is marked as private. We can change the current API then because my major concern was changing a public API. I will update new patch as per your suggestion then.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Varun Saxena,
        SchedulerUtils itself is marked as @private so you can modify it without any issue.
        I think the new method you added to SchedulerUtils is duplicated logic, could you add QueueInfo instead of creating new method?

        Show
        leftnoteasy Wangda Tan added a comment - Varun Saxena , SchedulerUtils itself is marked as @private so you can modify it without any issue. I think the new method you added to SchedulerUtils is duplicated logic, could you add QueueInfo instead of creating new method?
        Hide
        varun_saxena Varun Saxena added a comment -

        Its quite similar to what I have done. Added a new queueInfo parameter in a new API.
        I did not change the old API SchedulerUtils#normalizeAndvalidateRequest (which is without queueInfo) because it is marked as public.

        Show
        varun_saxena Varun Saxena added a comment - Its quite similar to what I have done. Added a new queueInfo parameter in a new API. I did not change the old API SchedulerUtils#normalizeAndvalidateRequest (which is without queueInfo) because it is marked as public.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Varun Saxena,
        I think we should pass queueinfo only once for a list of resource request.
        When RMServerUtils.normalizeAndValidateRequests is called, all ResourceRequests from same application, so they will from same queue as well. Then you can pass it down to normalizeAndvalidateRequest, In

          public static void normalizeAndValidateRequest(ResourceRequest resReq,
              Resource maximumResource, String queueName, YarnScheduler scheduler,
              boolean isRecovery)
        

        You can add a queue info as parameter, only get queue info from scheduler when it's not null. Thoughts?

        Show
        leftnoteasy Wangda Tan added a comment - Varun Saxena , I think we should pass queueinfo only once for a list of resource request. When RMServerUtils.normalizeAndValidateRequests is called, all ResourceRequests from same application, so they will from same queue as well. Then you can pass it down to normalizeAndvalidateRequest , In public static void normalizeAndValidateRequest(ResourceRequest resReq, Resource maximumResource, String queueName, YarnScheduler scheduler, boolean isRecovery) You can add a queue info as parameter, only get queue info from scheduler when it's not null. Thoughts?
        Hide
        kasha Karthik Kambatla added a comment -

        The patch doesn't apply anymore.

        Show
        kasha Karthik Kambatla added a comment - The patch doesn't apply anymore.
        Hide
        varun_saxena Varun Saxena added a comment -

        Vinod Kumar Vavilapalli, as the current API is called in places where there is only one resource request, hence queue info need not be passed.
        Some sort of API change will be required to pass the queue info only once for a list of resource requests. Thoughts ?

        Show
        varun_saxena Varun Saxena added a comment - Vinod Kumar Vavilapalli , as the current API is called in places where there is only one resource request, hence queue info need not be passed. Some sort of API change will be required to pass the queue info only once for a list of resource requests. Thoughts ?
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Both the classes are internal, you should change existing APIs instead of adding new ones. Also, if possible, cleanup other existing APIs too to avoid similar problems.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Both the classes are internal, you should change existing APIs instead of adding new ones. Also, if possible, cleanup other existing APIs too to avoid similar problems.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12726790/YARN-3489.02.patch
        against trunk revision d52de61.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7417//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7417//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726790/YARN-3489.02.patch against trunk revision d52de61. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7417//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7417//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12726511/YARN-3489.01.patch
        against trunk revision 8511d80.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7394//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12726511/YARN-3489.01.patch against trunk revision 8511d80. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7394//console This message is automatically generated.

          People

          • Assignee:
            varun_saxena Varun Saxena
            Reporter:
            jlowe Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development