Hadoop Common
  1. Hadoop Common
  2. HADOOP-5269

TaskTracker.runningTasks holding FAILED_UNCLEAN and KILLED_UNCLEAN taskStatuses forever in some cases.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.1
    • Fix Version/s: 0.19.2, 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Tasktracker is holdingup TaskStatus objects in runningTasks forever in somecases. This happens in the following scenario.
      -> Task got an exception
      -> Sets the phase to CLEANUP
      -> The task tries to do cleanup. and it doesn't respond after that.
      -> TaskTracker marks the task unresponsive and makes the task FAILED_UNCLEAN
      -> TaskTracker doesn't remove it from runningTasks data structure, since phase is CLEANUP and state is FAILED_UNCLEAN (it treats this as cleanupAttempt).

      I would propose that once the task goes to CLEANUP phase, kill on the task should mark it a clean failure i.e. The task state should be FAILED/KILLED.

      1. patch-5269-0.19-0.20.txt
        10 kB
        Amareshwari Sriramadasu
      2. patch-5269.txt
        11 kB
        Amareshwari Sriramadasu

        Activity

        Amareshwari Sriramadasu created issue -
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Fix Version/s 0.20.0 [ 12313438 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching patch with the fix.

        Show
        Amareshwari Sriramadasu added a comment - Attaching patch with the fix.
        Amareshwari Sriramadasu made changes -
        Attachment patch-5269.txt [ 12400577 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        test-patch result :

         
             [exec]
             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
        
        Show
        Amareshwari Sriramadasu added a comment - test-patch result : [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        ant tests passed on my machine.
        Ran Reliability test and Sort benchmark.
        Also verified OutofMemory run on which Vinod saw this issue.

        Show
        Amareshwari Sriramadasu added a comment - ant tests passed on my machine. Ran Reliability test and Sort benchmark. Also verified OutofMemory run on which Vinod saw this issue.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for 0.19 and 0.20

        Show
        Amareshwari Sriramadasu added a comment - Patch for 0.19 and 0.20
        Amareshwari Sriramadasu made changes -
        Attachment patch-5269-0.19-0.20.txt [ 12400585 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-5269-0.19-0.20.txt [ 12400585 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for 0.19 and 0.20. Earlier patch had an unnecessary comment... removed that.

        Show
        Amareshwari Sriramadasu added a comment - Patch for 0.19 and 0.20. Earlier patch had an unnecessary comment... removed that.
        Amareshwari Sriramadasu made changes -
        Attachment patch-5269-0.19-0.20.txt [ 12400587 ]
        Hide
        Devaraj Das added a comment -

        I just committed this to the 0.20 branch and trunk. Thanks, Amareshwari! (After 0.19.1 is released for which voting is going on, we should commit this to 0.19 branch as well)

        Show
        Devaraj Das added a comment - I just committed this to the 0.20 branch and trunk. Thanks, Amareshwari! (After 0.19.1 is released for which voting is going on, we should commit this to 0.19 branch as well)
        Devaraj Das made changes -
        Fix Version/s 0.19.1 [ 12313473 ]
        Hadoop Flags [Reviewed]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.21.0 [ 12313563 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #763 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/763/ )
        Amareshwari Sriramadasu made changes -
        Fix Version/s 0.19.2 [ 12313650 ]
        Hide
        Devaraj Das added a comment -

        I committed this to the 0.19 branch.

        Show
        Devaraj Das added a comment - I committed this to the 0.19 branch.
        Nigel Daley made changes -
        Fix Version/s 0.21.0 [ 12313563 ]
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        3d 52m 1 Amareshwari Sriramadasu 20/Feb/09 06:27
        Patch Available Patch Available Resolved Resolved
        3h 56m 1 Devaraj Das 20/Feb/09 10:24
        Resolved Resolved Closed Closed
        62d 8h 53m 1 Nigel Daley 23/Apr/09 20:18

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development