Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add following additional job tracker metrics:
      Reserved{Map, Reduce}Slots
      Occupied{Map, Reduce}Slots
      Running{Map, Reduce}Tasks
      Killed{Map, Reduce}Tasks

      FailedJobs
      KilledJobs
      PrepJobs
      RunningJobs

      TotalTrackers
      BlacklistedTrackers
      DecommissionedTrackers
      Show
      Add following additional job tracker metrics: Reserved{Map, Reduce}Slots Occupied{Map, Reduce}Slots Running{Map, Reduce}Tasks Killed{Map, Reduce}Tasks FailedJobs KilledJobs PrepJobs RunningJobs TotalTrackers BlacklistedTrackers DecommissionedTrackers

      Description

      It would be useful for tracking the following additional JobTracker metrics:
      running

      {map|reduce}tasks
      busy{map|reduce}

      slots
      reserved

      {map|reduce}

      slots

      1. 1103_v5_yahoo_1.patch
        40 kB
        Sharad Agarwal
      2. 1103_v5.patch
        53 kB
        Sharad Agarwal
      3. 1103_v4.patch
        44 kB
        Sharad Agarwal
      4. 1103_v3.patch
        44 kB
        Sharad Agarwal
      5. 1103_v2.patch
        46 kB
        Sharad Agarwal
      6. 1103_v1.patch
        40 kB
        Sharad Agarwal
      7. 1103.patch
        25 kB
        Sharad Agarwal
      8. 1103.patch
        25 kB
        Sharad Agarwal

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #127 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/127/ )
        Hide
        Sharad Agarwal added a comment -

        Yahoo! 0.20 patch.

        Show
        Sharad Agarwal added a comment - Yahoo! 0.20 patch.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #95 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/95/)
        . Added more metrics to Jobtracker. Contributed by Sharad Agarwal.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #95 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/95/ ) . Added more metrics to Jobtracker. Contributed by Sharad Agarwal.
        Hide
        Sharad Agarwal added a comment -

        I just committed this.

        Show
        Sharad Agarwal added a comment - I just committed this.
        Hide
        Jothi Padmanabhan added a comment -

        +1. Patch looks good

        Show
        Jothi Padmanabhan added a comment - +1. Patch looks good
        Hide
        Sharad Agarwal added a comment -

        Failure of TestGridmixSubmission seems unrelated. Filed MAPREDUCE-1132

        Show
        Sharad Agarwal added a comment - Failure of TestGridmixSubmission seems unrelated. Filed MAPREDUCE-1132
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12422879/1103_v5.patch
        against trunk revision 828253.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 30 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422879/1103_v5.patch against trunk revision 828253. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 30 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/200/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12422776/1103_v4.patch
        against trunk revision 827889.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422776/1103_v4.patch against trunk revision 827889. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/87/console This message is automatically generated.
        Hide
        Sharad Agarwal added a comment -

        Incorporated review comments.

        Show
        Sharad Agarwal added a comment - Incorporated review comments.
        Hide
        Jothi Padmanabhan added a comment -

        I forgot to add - the set of decommissionedTrackers is incorrect, it needs to be done outside of the for loop.

        Show
        Jothi Padmanabhan added a comment - I forgot to add - the set of decommissionedTrackers is incorrect, it needs to be done outside of the for loop.
        Hide
        Jothi Padmanabhan added a comment -

        Few points:

        1. Weighted methods could be removed from JobTrackerInstrumentation and JTMetricsInst
        2. JTMetricsInst.setMapSlots should also be synchronized
        3. Should we rename addBlacklistedTracker to addBlackListedTrackers, to be consistent with the other method names. Similarly for decBlackListedTracker, addTracker, decTracker
        4. There are some if else statements without the braces. I think we follow the style convention of having braces for if-else even if they are one line statements
        Show
        Jothi Padmanabhan added a comment - Few points: Weighted methods could be removed from JobTrackerInstrumentation and JTMetricsInst JTMetricsInst.setMapSlots should also be synchronized Should we rename addBlacklistedTracker to addBlackListedTrackers, to be consistent with the other method names. Similarly for decBlackListedTracker, addTracker, decTracker There are some if else statements without the braces. I think we follow the style convention of having braces for if-else even if they are one line statements
        Hide
        Sharad Agarwal added a comment -

        Fixed reserve slots metrics.

        Show
        Sharad Agarwal added a comment - Fixed reserve slots metrics.
        Hide
        Sharad Agarwal added a comment -

        Updated the test case.

        Show
        Sharad Agarwal added a comment - Updated the test case.
        Hide
        Sharad Agarwal added a comment -

        Removed the metrics related to High ram jobs
        Added test case. Thanks to Jothi for helping me with the test case.

        Show
        Sharad Agarwal added a comment - Removed the metrics related to High ram jobs Added test case. Thanks to Jothi for helping me with the test case.
        Hide
        Sharad Agarwal added a comment -

        Updated to trunk

        Show
        Sharad Agarwal added a comment - Updated to trunk
        Hide
        Sharad Agarwal added a comment -

        Adds following metrics:
        Reserved

        {Map, Reduce}Slots
        Occupied{Map, Reduce}

        Slots
        Running

        {Map, Reduce}Tasks
        Killed{Map, Reduce}

        Tasks

        FailedJobs
        KilledJobs
        PrepJobs
        RunningJobs

        TotalTrackers
        BlacklistedTrackers
        DecommissionedTrackers

        High RAM
        ---------------
        Pending

        {Map, Reduce}WeightedTasks
        Running{Map, Reduce}

        WeightedTasks
        Launched

        {Map, Reduce}WeightedTasks
        Completed{Map, Reduce}

        WeightedTasks
        Failed

        {Map, Reduce}WeightedTasks
        Killed{Map, Reduce}

        WeightedTasks

        Show
        Sharad Agarwal added a comment - Adds following metrics: Reserved {Map, Reduce}Slots Occupied{Map, Reduce} Slots Running {Map, Reduce}Tasks Killed{Map, Reduce} Tasks FailedJobs KilledJobs PrepJobs RunningJobs TotalTrackers BlacklistedTrackers DecommissionedTrackers High RAM --------------- Pending {Map, Reduce}WeightedTasks Running{Map, Reduce} WeightedTasks Launched {Map, Reduce}WeightedTasks Completed{Map, Reduce} WeightedTasks Failed {Map, Reduce}WeightedTasks Killed{Map, Reduce} WeightedTasks
        Hide
        Hong Tang added a comment -

        +1

        Show
        Hong Tang added a comment - +1

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Arun C Murthy
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development