Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: contrib/fair-share
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be useful to be able to monitor various metrics in the fair scheduler, like demand, fair share, min share, and running task count.

      1. mr-2323-20.txt
        12 kB
        Todd Lipcon
      2. mr-2323.txt
        21 kB
        Todd Lipcon
      3. mr-2323.txt
        23 kB
        Todd Lipcon
      4. mr-2323.txt
        27 kB
        Todd Lipcon

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4h 37m 1 Todd Lipcon 13/Feb/11 07:57
        Patch Available Patch Available Resolved Resolved
        142d 21h 14m 1 Todd Lipcon 06/Jul/11 06:12
        Resolved Resolved Closed Closed
        131d 19h 36m 1 Arun C Murthy 15/Nov/11 00:48
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #729 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/729/)
        MAPREDUCE-2323. Add metrics to the fair scheduler. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252
        Files :

        • /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #729 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/729/ ) MAPREDUCE-2323 . Add metrics to the fair scheduler. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252 Files : /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #734 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/734/)
        MAPREDUCE-2323. Add metrics to the fair scheduler. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252
        Files :

        • /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        • /hadoop/common/trunk/mapreduce/CHANGES.txt
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java
        • /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #734 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/734/ ) MAPREDUCE-2323 . Add metrics to the fair scheduler. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143252 Files : /hadoop/common/trunk/mapreduce/src/contrib/build-contrib.xml /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolSchedulable.java /hadoop/common/trunk/mapreduce/src/docs/src/documentation/content/xdocs/fair_scheduler.xml /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Pool.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/Schedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/FairScheduler.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/JobSchedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/TestFairScheduler.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/test/org/apache/hadoop/mapred/FakeSchedulable.java /hadoop/common/trunk/mapreduce/src/contrib/fairscheduler/src/java/org/apache/hadoop/mapred/PoolManager.java
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Todd Lipcon added a comment -

        Committed to trunk. Thanks for reviewing.

        Show
        Todd Lipcon added a comment - Committed to trunk. Thanks for reviewing.
        Hide
        Scott Chen added a comment -

        +1 The patch looks good to me.
        I think the failed tasks are unrelated.

        Show
        Scott Chen added a comment - +1 The patch looks good to me. I think the failed tasks are unrelated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12483015/mr-2323.txt
        against trunk revision 1137017.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.cli.TestMRCLI
        org.apache.hadoop.fs.TestFileSystem

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483015/mr-2323.txt against trunk revision 1137017. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/406//console This message is automatically generated.
        Todd Lipcon made changes -
        Attachment mr-2323.txt [ 12483015 ]
        Hide
        Todd Lipcon added a comment -

        Rebased on trunk. Also fixes a bug where using "setPool" twice on a job would cause an NPE, since it would clean up the metrics on the first switch, and then NPE when cleaning it up again

        Show
        Todd Lipcon added a comment - Rebased on trunk. Also fixes a bug where using "setPool" twice on a job would cause an NPE, since it would clean up the metrics on the first switch, and then NPE when cleaning it up again
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472627/mr-2323.txt
        against trunk revision 1076804.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472627/mr-2323.txt against trunk revision 1076804. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//console This message is automatically generated.
        Todd Lipcon made changes -
        Attachment mr-2323.txt [ 12472627 ]
        Hide
        Todd Lipcon added a comment -

        Slightly improved patch - the main code is the same, but this fixes some flaky behavior in the tests. I previously wasn't unregistering the metrics updater in fairsched's termination method, so different test cases were interacting poorly with eachother.

        This should be ready for review.

        Show
        Todd Lipcon added a comment - Slightly improved patch - the main code is the same, but this fixes some flaky behavior in the tests. I previously wasn't unregistering the metrics updater in fairsched's termination method, so different test cases were interacting poorly with eachother. This should be ready for review.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12470959/mr-2323.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.mapred.TestDebugScript

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12470959/mr-2323.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestDebugScript -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/30//console This message is automatically generated.
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Todd Lipcon made changes -
        Attachment mr-2323.txt [ 12470959 ]
        Hide
        Todd Lipcon added a comment -

        Here's a patch against trunk with unit tests and docs. Above questions still stand.

        Show
        Todd Lipcon added a comment - Here's a patch against trunk with unit tests and docs. Above questions still stand.
        Hide
        Todd Lipcon added a comment -

        Would appreciate feedback on the following:

        • is the locking correct for updateMetrics? I take a lock on the scheduler and then the poolmanager, but no other locks.
        • could it be finer grain? (do I need the coarse scheduler lock at all? unclear)
        • should I add a configuration to disable the job-level metrics and only use pool-level metrics?
        • when a user pool becomes empty, should we remove its metrics records?
        Show
        Todd Lipcon added a comment - Would appreciate feedback on the following: is the locking correct for updateMetrics? I take a lock on the scheduler and then the poolmanager, but no other locks. could it be finer grain? (do I need the coarse scheduler lock at all? unclear) should I add a configuration to disable the job-level metrics and only use pool-level metrics? when a user pool becomes empty, should we remove its metrics records?
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment mr-2323-20.txt [ 12470958 ]
        Hide
        Todd Lipcon added a comment -

        Here's a quick draft of this feature against our branch 20. Will work on updating to trunk and adding some kind of tests.

        Show
        Todd Lipcon added a comment - Here's a quick draft of this feature against our branch 20. Will work on updating to trunk and adding some kind of tests.
        Todd Lipcon created issue -

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development