Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6042

Dump scheduler and queue state information into FairScheduler DEBUG log

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fairscheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      To improve the debugging of scheduler issues it would be a big improvement to be able to dump the scheduler state into a log on request.
      The Dump the scheduler state at a point in time would allow debugging of a scheduler that is not hung (deadlocked) but also not assigning containers. Currently we do not have a proper overview of what state the scheduler and the queues are in and we have to make assumptions or guess
      The scheduler and queue state needed would include (not exhaustive):

      • instantaneous and steady fair share (app / queue)
      • AM share and resources
      • weight
      • app demand
      • application run state (runnable/non runnable)
      • last time at fair/min share
      1. YARN-6042.001.patch
        11 kB
        Yufei Gu
      2. YARN-6042.002.patch
        12 kB
        Yufei Gu
      3. YARN-6042.003.patch
        16 kB
        Yufei Gu
      4. YARN-6042.004.patch
        16 kB
        Yufei Gu
      5. YARN-6042.005.patch
        16 kB
        Yufei Gu
      6. YARN-6042.006.patch
        16 kB
        Yufei Gu
      7. YARN-6042.007.patch
        16 kB
        Yufei Gu
      8. YARN-6042.008.patch
        18 kB
        Yufei Gu
      9. YARN-6042.009.patch
        18 kB
        Yufei Gu
      10. YARN-6042.010.patch
        18 kB
        Yufei Gu
      11. YARN-6042.branch-2.001.patch
        18 kB
        Yufei Gu

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 41s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 37s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 12s trunk passed
          +1 javadoc 0m 25s trunk passed
          +1 mvninstall 0m 39s the patch passed
          +1 compile 0m 39s the patch passed
          +1 javac 0m 39s the patch passed
          -0 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 262 unchanged - 0 fixed = 263 total (was 262)
          +1 mvnsite 0m 38s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 22s the patch passed
          +1 javadoc 0m 23s the patch passed
          +1 unit 42m 9s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          65m 35s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850983/YARN-6042.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0e1c6c7dfcfe 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3ea6d35
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14827/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14827/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14827/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 41s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 0m 39s the patch passed +1 javac 0m 39s the patch passed -0 checkstyle 0m 25s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 262 unchanged - 0 fixed = 263 total (was 262) +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 22s the patch passed +1 javadoc 0m 23s the patch passed +1 unit 42m 9s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 65m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12850983/YARN-6042.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0e1c6c7dfcfe 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ea6d35 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14827/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14827/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14827/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          I know this is for debugging only, but if you have a lot of deep queue hierarchies, you'll have a lot of unnecessary StringBuilder->String conversions at each FSParentQueue. I think it would be better to keep things as StringBuilder throughout the call hierarchy and do the conversion to String at the end in FSAppAttempt.

          Show
          rchiang Ray Chiang added a comment - I know this is for debugging only, but if you have a lot of deep queue hierarchies, you'll have a lot of unnecessary StringBuilder->String conversions at each FSParentQueue. I think it would be better to keep things as StringBuilder throughout the call hierarchy and do the conversion to String at the end in FSAppAttempt.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray Chiang for the review. That's a fair point. I've uploaded a new patch for that. Additionally I add queue policy in the state dump.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray Chiang for the review. That's a fair point. I've uploaded a new patch for that. Additionally I add queue policy in the state dump.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 55s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 40s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 15s trunk passed
          +1 javadoc 0m 25s trunk passed
          +1 mvninstall 0m 35s the patch passed
          +1 compile 0m 34s the patch passed
          +1 javac 0m 34s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 38s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 18s the patch passed
          +1 javadoc 0m 22s the patch passed
          -1 unit 42m 38s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          67m 20s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851257/YARN-6042.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a630248e21f7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d88497d
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14836/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14836/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14836/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 55s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 15s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 35s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 42m 38s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 67m 20s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12851257/YARN-6042.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a630248e21f7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d88497d Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14836/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14836/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14836/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          I looked at the changes and it will help debugging the FS a lot when we get this into a release

          A couple of things:

          1. In the FairScheduler change you add a new method dumpSchedulerState() why are you not passing in the rootQueue to the method? It safes getting it again since you have already got it the update method.
          2. I am missing one number for the applications in the dumpStateInternal() for the FSLeafQueue: getNumPendingApps() or getNumActiveApps(). We need to have one of those to have a full view of what the application state is in the queue.
          3. We add the LastTimeAtMinShare but not the LastTimeAtFairShare for the leaf queue as per: getLastTimeAtFairShareThreshold()

          I am also a bit worried about the test: in the output we build the debug string and get the time in milliseconds for the LastTimeAtMinShare. What if the updateStarvationStats() call was run 1 millisecond earlier than the debug string was build? The comparison would fail and the test would fail because of that. I don't think we can guarantee that those two calls will be in the same millisecond.

          Show
          wilfreds Wilfred Spiegelenburg added a comment - I looked at the changes and it will help debugging the FS a lot when we get this into a release A couple of things: In the FairScheduler change you add a new method dumpSchedulerState() why are you not passing in the rootQueue to the method? It safes getting it again since you have already got it the update method. I am missing one number for the applications in the dumpStateInternal() for the FSLeafQueue: getNumPendingApps() or getNumActiveApps() . We need to have one of those to have a full view of what the application state is in the queue. We add the LastTimeAtMinShare but not the LastTimeAtFairShare for the leaf queue as per: getLastTimeAtFairShareThreshold() I am also a bit worried about the test: in the output we build the debug string and get the time in milliseconds for the LastTimeAtMinShare. What if the updateStarvationStats() call was run 1 millisecond earlier than the debug string was build? The comparison would fail and the test would fail because of that. I don't think we can guarantee that those two calls will be in the same millisecond.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Wilfred Spiegelenburg's review.
          1. Pass rootQueue to the method doesn't change much since rootQueue is a member of QueueManager
          2. Fixed in new patch.
          3. Some changes in trunk removed getLastTimeAtFairShareThreshold()
          4. ControlledClock is used in test case, so the clock won't run unless we manually tick it.
          5. As our offline discussion, I add more log into several steps of assigning containers to resource requests.

          Show
          yufeigu Yufei Gu added a comment - Thanks Wilfred Spiegelenburg 's review. 1. Pass rootQueue to the method doesn't change much since rootQueue is a member of QueueManager 2. Fixed in new patch. 3. Some changes in trunk removed getLastTimeAtFairShareThreshold() 4. ControlledClock is used in test case, so the clock won't run unless we manually tick it. 5. As our offline discussion, I add more log into several steps of assigning containers to resource requests.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user flyrain opened a pull request:

          https://github.com/apache/hadoop/pull/193

          YARN-6042. Fairscheduler: Dump scheduler state in log.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/flyrain/hadoop yarn-6042

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/193.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #193


          commit de6c969973d5b50aa2b461be1560b1e9b80cc9dc
          Author: Yufei Gu <yufei.gu@cloudera.com>
          Date: 2017-01-13T01:35:17Z

          YARN-6042. Fairscheduler: Dump scheduler state in log.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user flyrain opened a pull request: https://github.com/apache/hadoop/pull/193 YARN-6042 . Fairscheduler: Dump scheduler state in log. You can merge this pull request into a Git repository by running: $ git pull https://github.com/flyrain/hadoop yarn-6042 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/193.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #193 commit de6c969973d5b50aa2b461be1560b1e9b80cc9dc Author: Yufei Gu <yufei.gu@cloudera.com> Date: 2017-01-13T01:35:17Z YARN-6042 . Fairscheduler: Dump scheduler state in log.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 16m 27s trunk passed
          +1 compile 0m 37s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 39s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 8s trunk passed
          +1 javadoc 0m 24s trunk passed
          +1 mvninstall 0m 35s the patch passed
          +1 compile 0m 32s the patch passed
          +1 javac 0m 32s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 34s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 16s the patch passed
          +1 javadoc 0m 20s the patch passed
          -1 unit 42m 42s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          68m 44s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cdd33a269f98 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0cf5993
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14940/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14940/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14940/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 16m 27s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 8s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 35s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 20s the patch passed -1 unit 42m 42s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 68m 44s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cdd33a269f98 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cf5993 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14940/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14940/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14940/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Patch v4 fixed the unit test.

          Show
          yufeigu Yufei Gu added a comment - Patch v4 fixed the unit test.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 11s YARN-6042 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14944/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 11s YARN-6042 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Console output https://builds.apache.org/job/PreCommit-YARN-Build/14944/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Patch v5 is a rebase. The upstream changes too fast.

          Show
          yufeigu Yufei Gu added a comment - Patch v5 is a rebase. The upstream changes too fast.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 28s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 46s trunk passed
          +1 compile 0m 32s trunk passed
          +1 checkstyle 0m 24s trunk passed
          +1 mvnsite 0m 33s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 57s trunk passed
          +1 javadoc 0m 21s trunk passed
          +1 mvninstall 0m 30s the patch passed
          +1 compile 0m 29s the patch passed
          +1 javac 0m 29s the patch passed
          +1 checkstyle 0m 21s the patch passed
          +1 mvnsite 0m 30s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 18s the patch passed
          +1 unit 42m 52s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          64m 12s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9df028d97374 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8e53f2b
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14947/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14947/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 28s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 46s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 33s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 29s the patch passed +1 javac 0m 29s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 30s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 42m 52s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 64m 12s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9df028d97374 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8e53f2b Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14947/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14947/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          wilfreds Wilfred Spiegelenburg added a comment -

          Thank you for the update, we discussed it offline and the change looks good now.

          I have one minor nit in the text update we should have in FSAppAttempt: creating should be create in the debug text
          LOG.debug("Couldn't creating reservation for app: " + getName()

          Show
          wilfreds Wilfred Spiegelenburg added a comment - Thank you for the update, we discussed it offline and the change looks good now. I have one minor nit in the text update we should have in FSAppAttempt: creating should be create in the debug text LOG.debug("Couldn't creating reservation for app: " + getName()
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Wilfred Spiegelenburg for the review. Fixed the issue in patch v6.

          Show
          yufeigu Yufei Gu added a comment - Thanks Wilfred Spiegelenburg for the review. Fixed the issue in patch v6.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 4s trunk passed
          +1 compile 0m 40s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 43s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 9s trunk passed
          +1 javadoc 0m 24s trunk passed
          +1 mvninstall 0m 38s the patch passed
          +1 compile 0m 36s the patch passed
          +1 javac 0m 36s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 37s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 14s the patch passed
          +1 javadoc 0m 19s the patch passed
          -1 unit 44m 8s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          68m 58s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4cac1b7941d3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 627da6f
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/14966/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14966/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14966/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 4s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 43s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 0m 36s the patch passed +1 javac 0m 36s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 14s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 44m 8s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 68m 58s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4cac1b7941d3 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 627da6f Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/14966/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14966/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14966/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          The test failures are unrelated. Ray Chiang, wanna take a look?

          Show
          yufeigu Yufei Gu added a comment - The test failures are unrelated. Ray Chiang , wanna take a look?
          Hide
          rchiang Ray Chiang added a comment -

          Looks like YARN-4212 means this patch no longer applies cleanly.

          Show
          rchiang Ray Chiang added a comment - Looks like YARN-4212 means this patch no longer applies cleanly.
          Hide
          yufeigu Yufei Gu added a comment -

          Rebase it in patch v7.

          Show
          yufeigu Yufei Gu added a comment - Rebase it in patch v7.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 9s YARN-6042 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/14993/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 9s YARN-6042 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Console output https://builds.apache.org/job/PreCommit-YARN-Build/14993/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          yufeigu Yufei Gu added a comment -

          Patch v7 has rebased, but PR doesn't. Update PR as well.

          Show
          yufeigu Yufei Gu added a comment - Patch v7 has rebased, but PR doesn't. Update PR as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 51s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 19m 32s trunk passed
          +1 compile 0m 51s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 22s trunk passed
          +1 findbugs 1m 28s trunk passed
          +1 javadoc 0m 35s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 45s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 35s the patch passed
          +1 javadoc 0m 29s the patch passed
          -1 unit 57m 26s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          89m 53s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 7184aaafdbfc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4c26c24
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15005/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15005/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15005/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 51s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 19m 32s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 22s trunk passed +1 findbugs 1m 28s trunk passed +1 javadoc 0m 35s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 45s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 35s the patch passed +1 javadoc 0m 29s the patch passed -1 unit 57m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 89m 53s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7184aaafdbfc 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4c26c24 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15005/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15005/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15005/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Very minor nit:

          The result of this part of code:

          rootMetrics.getAvailableMB(), rootMetrics.getAvailableVirtualCores()) +
          rootQueue.dumpState());

          There is no separation between the scheduler and the queue states. From my sample output, the part in red looks a little odd:

          2017-02-23 14:53:29,644 DEBUG fair.FairScheduler: FairScheduler state: Cluster Capacity: <memory:0, vCores:0> Allocations: <memory:0, vCores:0> Availability: <memory:0, vCores:0>{Name: root, Weight: <memory weight=1.0, cpu weight=1.0>, Policy: fair, FairShare: <memory:0, vCores:0>, SteadyFairShare: <memory:0, vCores:0>,

          I'd suggest adding two spaces and possibly a label like the rest of the scheduler state?

          Show
          rchiang Ray Chiang added a comment - Very minor nit: The result of this part of code: rootMetrics.getAvailableMB(), rootMetrics.getAvailableVirtualCores()) + rootQueue.dumpState()); There is no separation between the scheduler and the queue states. From my sample output, the part in red looks a little odd: 2017-02-23 14:53:29,644 DEBUG fair.FairScheduler: FairScheduler state: Cluster Capacity: <memory:0, vCores:0> Allocations: <memory:0, vCores:0> Availability: <memory:0, vCores:0 >{ Name: root, Weight: <memory weight=1.0, cpu weight=1.0>, Policy: fair, FairShare: <memory:0, vCores:0>, SteadyFairShare: <memory:0, vCores:0>, I'd suggest adding two spaces and possibly a label like the rest of the scheduler state?
          Hide
          rchiang Ray Chiang added a comment -

          After an offline conversation with Daniel Templeton, it looks like it would be a good idea to put these queue updates pointing at a separate log4j property and make the default a separate file. That should help with the huge volume of information this debug setting will put in a log file.

          Show
          rchiang Ray Chiang added a comment - After an offline conversation with Daniel Templeton , it looks like it would be a good idea to put these queue updates pointing at a separate log4j property and make the default a separate file. That should help with the huge volume of information this debug setting will put in a log file.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray Chiang for the review. Uploaded patch v8 for all your comments.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray Chiang for the review. Uploaded patch v8 for all your comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 48s Maven dependency ordering for branch
          +1 mvninstall 14m 11s trunk passed
          +1 compile 18m 57s trunk passed
          +1 checkstyle 2m 8s trunk passed
          +1 mvnsite 2m 13s trunk passed
          +1 mvneclipse 0m 42s trunk passed
          +1 findbugs 3m 4s trunk passed
          +1 javadoc 1m 25s trunk passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 24s the patch passed
          +1 compile 14m 22s the patch passed
          +1 javac 14m 22s the patch passed
          +1 checkstyle 2m 6s the patch passed
          +1 mvnsite 1m 59s the patch passed
          +1 mvneclipse 0m 46s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 53s the patch passed
          +1 javadoc 1m 35s the patch passed
          -1 unit 9m 3s hadoop-common in the patch failed.
          -1 unit 42m 24s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          147m 29s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin
            hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
            hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle
          uname Linux 4353b8e1e9b7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 480b4dd
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15107/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15107/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15107/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15107/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 48s Maven dependency ordering for branch +1 mvninstall 14m 11s trunk passed +1 compile 18m 57s trunk passed +1 checkstyle 2m 8s trunk passed +1 mvnsite 2m 13s trunk passed +1 mvneclipse 0m 42s trunk passed +1 findbugs 3m 4s trunk passed +1 javadoc 1m 25s trunk passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 24s the patch passed +1 compile 14m 22s the patch passed +1 javac 14m 22s the patch passed +1 checkstyle 2m 6s the patch passed +1 mvnsite 1m 59s the patch passed +1 mvneclipse 0m 46s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 53s the patch passed +1 javadoc 1m 35s the patch passed -1 unit 9m 3s hadoop-common in the patch failed. -1 unit 42m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 147m 29s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin   hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart   hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle uname Linux 4353b8e1e9b7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 480b4dd Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15107/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15107/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15107/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15107/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          Tao Jie Tao Jie added a comment -

          Hi Yufei Gu, dumping scheduler/queue state is very useful to detect scheduling problem at run-time. It seems to me that you try write scheduler/queue information to log file. How about print this information on the webui, just like we can get server stacks by a link.

          Show
          Tao Jie Tao Jie added a comment - Hi Yufei Gu , dumping scheduler/queue state is very useful to detect scheduling problem at run-time. It seems to me that you try write scheduler/queue information to log file. How about print this information on the webui, just like we can get server stacks by a link.
          Hide
          yufeigu Yufei Gu added a comment - - edited

          Tao Jie, YARN-5437 is an umbrella which adds useful messages of scheduler in WebUI. YARN-4329 is the FS part, but we can always add more useful information in WebUI. Not a bad idea to keep improvement on that. Other than that, we need add more queue metrics on scheduler WebUI, which I cannot remember a JIRA for that. This JIRA will potentially dump a very long message, there has been a link for the RM log file in WebUI, and adding a link for the new log file may be a reasonable solution.

          Show
          yufeigu Yufei Gu added a comment - - edited Tao Jie , YARN-5437 is an umbrella which adds useful messages of scheduler in WebUI. YARN-4329 is the FS part, but we can always add more useful information in WebUI. Not a bad idea to keep improvement on that. Other than that, we need add more queue metrics on scheduler WebUI, which I cannot remember a JIRA for that. This JIRA will potentially dump a very long message, there has been a link for the RM log file in WebUI, and adding a link for the new log file may be a reasonable solution.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks for the review Ray Chiang, uploaded patch v9 to get logger name correctly.
          BTW, Tao Jie, continue the previous discussion actually no need to add a new link, you can find the new log file in RM webUI by going to "tool" -> "Local Logs"

          Show
          yufeigu Yufei Gu added a comment - Thanks for the review Ray Chiang , uploaded patch v9 to get logger name correctly. BTW, Tao Jie , continue the previous discussion actually no need to add a new link, you can find the new log file in RM webUI by going to "tool" -> "Local Logs"
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 57s Maven dependency ordering for branch
          +1 mvninstall 14m 2s trunk passed
          +1 compile 10m 45s trunk passed
          +1 checkstyle 2m 3s trunk passed
          +1 mvnsite 1m 53s trunk passed
          +1 mvneclipse 0m 51s trunk passed
          +1 findbugs 2m 45s trunk passed
          +1 javadoc 1m 31s trunk passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 12s the patch passed
          +1 compile 10m 10s the patch passed
          +1 javac 10m 10s the patch passed
          +1 checkstyle 2m 4s the patch passed
          +1 mvnsite 1m 56s the patch passed
          +1 mvneclipse 0m 57s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 5s the patch passed
          +1 javadoc 1m 35s the patch passed
          -1 unit 9m 8s hadoop-common in the patch failed.
          +1 unit 40m 53s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 55s The patch does not generate ASF License warnings.
          133m 21s



          Reason Tests
          Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle
          uname Linux a23fc4b1c24c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5e74196
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15185/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15185/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15185/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 57s Maven dependency ordering for branch +1 mvninstall 14m 2s trunk passed +1 compile 10m 45s trunk passed +1 checkstyle 2m 3s trunk passed +1 mvnsite 1m 53s trunk passed +1 mvneclipse 0m 51s trunk passed +1 findbugs 2m 45s trunk passed +1 javadoc 1m 31s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 10m 10s the patch passed +1 javac 10m 10s the patch passed +1 checkstyle 2m 4s the patch passed +1 mvnsite 1m 56s the patch passed +1 mvneclipse 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 5s the patch passed +1 javadoc 1m 35s the patch passed -1 unit 9m 8s hadoop-common in the patch failed. +1 unit 40m 53s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 55s The patch does not generate ASF License warnings. 133m 21s Reason Tests Failed junit tests hadoop.fs.sftp.TestSFTPFileSystem Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle uname Linux a23fc4b1c24c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5e74196 Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/15185/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15185/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15185/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Looks good so far. Another things occurs to me. The queue info is being dumped out every 3 seconds on my machine. Can we add a configuration option (log4j or yarn-default.xml) that would lower the output rate of these messages? I think an update every 15-30 seconds would be enough in many cases.

          Show
          rchiang Ray Chiang added a comment - Looks good so far. Another things occurs to me. The queue info is being dumped out every 3 seconds on my machine. Can we add a configuration option (log4j or yarn-default.xml) that would lower the output rate of these messages? I think an update every 15-30 seconds would be enough in many cases.
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray Chiang for the review. Right now the default update interval is 0.5s. And we already have UPDATE_DEBUG_FREQUENCY to output debug message less frequently. Right now UPDATE_DEBUG_FREQUENCY is 5, which means every 0.5 * 5 s, there is a state dump. I increased it to 25 in patch v10. So FS dumps its state every 12.5s.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray Chiang for the review. Right now the default update interval is 0.5s. And we already have UPDATE_DEBUG_FREQUENCY to output debug message less frequently. Right now UPDATE_DEBUG_FREQUENCY is 5, which means every 0.5 * 5 s, there is a state dump. I increased it to 25 in patch v10. So FS dumps its state every 12.5s.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 30s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 58s Maven dependency ordering for branch
          +1 mvninstall 12m 47s trunk passed
          +1 compile 13m 30s trunk passed
          +1 checkstyle 2m 22s trunk passed
          +1 mvnsite 2m 5s trunk passed
          +1 mvneclipse 0m 50s trunk passed
          +1 findbugs 3m 1s trunk passed
          +1 javadoc 1m 33s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 13s the patch passed
          +1 compile 10m 20s the patch passed
          +1 javac 10m 20s the patch passed
          -0 checkstyle 4m 28s root: The patch generated 1 new + 255 unchanged - 1 fixed = 256 total (was 256)
          +1 mvnsite 3m 52s the patch passed
          +1 mvneclipse 0m 57s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 1m 40s the patch passed
          +1 unit 8m 45s hadoop-common in the patch passed.
          -1 unit 44m 39s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 47s The patch does not generate ASF License warnings.
          143m 51s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-6042
          GITHUB PR https://github.com/apache/hadoop/pull/193
          Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle
          uname Linux 24ca36814643 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 822a74f
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15219/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15219/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15219/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15219/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 30s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 58s Maven dependency ordering for branch +1 mvninstall 12m 47s trunk passed +1 compile 13m 30s trunk passed +1 checkstyle 2m 22s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 50s trunk passed +1 findbugs 3m 1s trunk passed +1 javadoc 1m 33s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 10m 20s the patch passed +1 javac 10m 20s the patch passed -0 checkstyle 4m 28s root: The patch generated 1 new + 255 unchanged - 1 fixed = 256 total (was 256) +1 mvnsite 3m 52s the patch passed +1 mvneclipse 0m 57s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 40s the patch passed +1 unit 8m 45s hadoop-common in the patch passed. -1 unit 44m 39s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 47s The patch does not generate ASF License warnings. 143m 51s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6042 GITHUB PR https://github.com/apache/hadoop/pull/193 Optional Tests asflicense mvnsite unit compile javac javadoc mvninstall findbugs checkstyle uname Linux 24ca36814643 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 822a74f Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15219/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15219/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15219/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15219/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          LGTM. +1

          Will commit this soon.

          Yufei Gu, please provide detailed information about how to turn this on in the Release Notes.

          Show
          rchiang Ray Chiang added a comment - LGTM. +1 Will commit this soon. Yufei Gu , please provide detailed information about how to turn this on in the Release Notes.
          Hide
          rchiang Ray Chiang added a comment -

          Committed to trunk. You'll need a different patch for branch-2, if you want.

          Thanks Yufei Gu for your contribution! Thanks Wilfred Spiegelenburg for reviewing!

          Show
          rchiang Ray Chiang added a comment - Committed to trunk. You'll need a different patch for branch-2, if you want. Thanks Yufei Gu for your contribution! Thanks Wilfred Spiegelenburg for reviewing!
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray Chiang for the review and commit. Thanks Wilfred Spiegelenburg for the review. Will post a branch-2 patch later.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray Chiang for the review and commit. Thanks Wilfred Spiegelenburg for the review. Will post a branch-2 patch later.
          Hide
          yufeigu Yufei Gu added a comment -

          Uploaded patch for branch2

          Show
          yufeigu Yufei Gu added a comment - Uploaded patch for branch2
          Hide
          rchiang Ray Chiang added a comment -

          Pushed to branch-2. Updated version to match. Thanks Yufei Gu.

          Show
          rchiang Ray Chiang added a comment - Pushed to branch-2. Updated version to match. Thanks Yufei Gu .
          Hide
          yufeigu Yufei Gu added a comment -

          Thanks Ray Chiang for the commit.

          Show
          yufeigu Yufei Gu added a comment - Thanks Ray Chiang for the commit.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user flyrain commented on the issue:

          https://github.com/apache/hadoop/pull/193

          Committed

          Show
          githubbot ASF GitHub Bot added a comment - Github user flyrain commented on the issue: https://github.com/apache/hadoop/pull/193 Committed
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user flyrain closed the pull request at:

          https://github.com/apache/hadoop/pull/193

          Show
          githubbot ASF GitHub Bot added a comment - Github user flyrain closed the pull request at: https://github.com/apache/hadoop/pull/193
          Hide
          aw Allen Wittenauer added a comment -

          FYI, I've opened YARN-6453 as a blocker due to some very bad behavior by this patch.

          Show
          aw Allen Wittenauer added a comment - FYI, I've opened YARN-6453 as a blocker due to some very bad behavior by this patch.

            People

            • Assignee:
              yufeigu Yufei Gu
              Reporter:
              yufeigu Yufei Gu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development