[MAPREDUCE-5512] TaskTracker hung after failed reconnect to the JobTracker

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1-win, 1.3.0
• Component/s: tasktracker
• Hadoop Flags: Reviewed
• Assignee: Ivan Mitic
• Reporter: Ivan Mitic
• Created: 17/Sep/13 01:53
• Updated: 11/Oct/13 05:57
• Resolved: 11/Oct/13 05:57

Description

TaskTracker hung after failed reconnect to the JobTracker.

This is the problematic piece of code:

    this.distributedCacheManager = new TrackerDistributedCacheManager(
        this.fConf, taskController);
    this.distributedCacheManager.startCleanupThread();
    
    this.jobClient = (InterTrackerProtocol) 
    UserGroupInformation.getLoginUser().doAs(
        new PrivilegedExceptionAction<Object>() {
      public Object run() throws IOException {
        return RPC.waitForProxy(InterTrackerProtocol.class,
            InterTrackerProtocol.versionID,
            jobTrackAddr, fConf);
      }
    });

In case RPC.waitForProxy() throws, TrackerDistributedCacheManager cleanup thread will never be stopped, and given that it is a non daemon thread it will keep TT up forever.