Failed/Killed attempts column in jobdetails.jsp does not show the number of failed/killed attempts correctly

• Key: HADOOP-5067
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Amareshwari Sriramadasu
• Reporter: Amareshwari Sriramadasu
• Affects Version/s: 0.19.0
• Fix Version/s: 0.19.1

Description

I see one of the task failures when i see it from the taskdetails.jsp page, but failed/killed attempts column show it as zero.

Comments

[Amareshwari Sriramadasu]

The failure was because of Too many fecth failures.

[Devaraj Das]

I noticed that even for Lost TT, the column doesn't list the KILLED maps. Maybe the problem is generally true for COMPLETED maps that are now KILLED.

[Amareshwari Sriramadasu]

This got introduced in 0.19 by HADOOP-3245

[Amar Kamat]

Looks like the problem is with

boolean isPresent = this.activeTasks.remove(taskid) != null;

This check is made to avoid wrong/incorrect updates for stray attempts (attempts for which there is no entry in the jobhistory). Consider a case where the attempt start and end lines are still in buffer when the jobtracker dies. In such a case the reducers might get the map completion event but the restarted jobtracker might not know about the attempt. So ideally any complaint about this map attempt should be ignored as the map will be re-executed. The idea was to update only if the TaskInProgress knows about the attempt. May be we should use tasks.keySet().contains() instead of activeTasks.remove(). Thoughts?

[Amareshwari Sriramadasu]

Attaching patch with the fix. Also added a test-case

[Amareshwari Sriramadasu]

test-patch result:

     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]

[Amareshwari Sriramadasu]

The patch applies to trunk and 0.20. I will upload a patch for 0.19.

[Amar Kamat]

Looks like the testcase is similar to TestJobTrackerRestartWithLostTracker. Can we factor out/reuse that code? The changes in the framework looks fine to me.

[Amareshwari Sriramadasu]

patch for branch 0.19

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12399641/patch-5067.txt
against trunk revision 741330.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3805/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3805/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3805/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3805/console

This message is automatically generated.

[Amareshwari Sriramadasu]

I dont see much scope for refactoring in the testcase. The main methods are already refactored into UtilsForTests class.

[Amareshwari Sriramadasu]

cancelling patch to optimize testcase

[Amareshwari Sriramadasu]

patch for 0.19

[Amareshwari Sriramadasu]

patch for trunk and 0.20

[Amareshwari Sriramadasu]

Runtime for the testcase:
[junit] Running org.apache.hadoop.mapred.TestLostTracker
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.852 sec

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12400077/patch-5067-1.txt
against trunk revision 743513.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3838/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3838/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3838/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3838/console

This message is automatically generated.

[Devaraj Das]

I just committed this. Thanks, Amareshwari!

[Hudson]

Integrated in Hadoop-trunk #756 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/756/)