Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1152

JobTrackerInstrumentation.killed{Map/Reduce} is never called

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      JobTrackerInstrumentation.killed

      {Map/Reduce}

      metrics added as part of MAPREDUCE-1103 is not captured

      1. 1152.patch
        3 kB
        Sharad Agarwal
      2. 1152.patch
        5 kB
        Sharad Agarwal
      3. 1152_v3.patch
        5 kB
        Sharad Agarwal
      4. 1152_v2.patch
        5 kB
        Sharad Agarwal

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #164 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/164/)
        . Distinguish between failed and killed tasks in
        JobTrackerInstrumentation. Contributed by Sharad Agarwal

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #164 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/164/ ) . Distinguish between failed and killed tasks in JobTrackerInstrumentation. Contributed by Sharad Agarwal
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #144 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/144/)
        . Distinguish between failed and killed tasks in
        JobTrackerInstrumentation. Contributed by Sharad Agarwal

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #144 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/144/ ) . Distinguish between failed and killed tasks in JobTrackerInstrumentation. Contributed by Sharad Agarwal
        Hide
        Chris Douglas added a comment -

        +1

        I committed this. Thanks, Sharad!

        Show
        Chris Douglas added a comment - +1 I committed this. Thanks, Sharad!
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423908/1152_v3.patch
        against trunk revision 832362.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423908/1152_v3.patch against trunk revision 832362. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/222/console This message is automatically generated.
        Hide
        Sharad Agarwal added a comment -

        Fixed findbugs warning.

        Show
        Sharad Agarwal added a comment - Fixed findbugs warning.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423812/1152_v2.patch
        against trunk revision 831816.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423812/1152_v2.patch against trunk revision 831816. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/120/console This message is automatically generated.
        Hide
        Sharad Agarwal added a comment -

        Updated to trunk.

        Show
        Sharad Agarwal added a comment - Updated to trunk.
        Hide
        Sharad Agarwal added a comment -

        Patch got stale.

        Show
        Sharad Agarwal added a comment - Patch got stale.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12423170/1152.patch
        against trunk revision 829529.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12423170/1152.patch against trunk revision 829529. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/212/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch looks fine to me.
        Submitting for hudson

        Show
        Amareshwari Sriramadasu added a comment - Patch looks fine to me. Submitting for hudson
        Hide
        Sharad Agarwal added a comment -

        Patch fixing fail and kill task metrics.

        Show
        Sharad Agarwal added a comment - Patch fixing fail and kill task metrics.
        Hide
        Arun C Murthy added a comment -

        Hmm... shouldn't we fix metrics to correctly record KILLED tasks separately from FAILED ones? The proposal in this jira seems like the wrong direction... we probably need JobCounter.NUM_KILLED_

        {MAPS|REDUCES}

        counters too?

        Show
        Arun C Murthy added a comment - Hmm... shouldn't we fix metrics to correctly record KILLED tasks separately from FAILED ones? The proposal in this jira seems like the wrong direction... we probably need JobCounter.NUM_KILLED_ {MAPS|REDUCES} counters too?
        Hide
        Sharad Agarwal added a comment -

        Straight forward patch.

        Show
        Sharad Agarwal added a comment - Straight forward patch.
        Hide
        Sharad Agarwal added a comment -

        Currently JobTrackerInstrumentation.failed

        {Map/Reduce} accounts for killed map/reduce tasks as well. In other places also like in JobCounter.NUM_FAILED_MAPS, killed and failed both maps are accounted. So I propose to remove the JobTrackerInstrumentation.killed{Map/Reduce}

        API since in framework we do the metering as part of failed API.

        Show
        Sharad Agarwal added a comment - Currently JobTrackerInstrumentation.failed {Map/Reduce} accounts for killed map/reduce tasks as well. In other places also like in JobCounter.NUM_FAILED_MAPS, killed and failed both maps are accounted. So I propose to remove the JobTrackerInstrumentation.killed{Map/Reduce} API since in framework we do the metering as part of failed API.

          People

          • Assignee:
            Unassigned
            Reporter:
            Sharad Agarwal
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development