The reason for the above exception is as follows:
- MemoryMatcher.getMemReservedForTasks tries to retrieve the JobConf for a job whose task has been reported in the TaskTrackerStatus in order to compute the available memory.
- This job has been retired and hence the lookup returns null, which is not checked.
- This scenario can occur if a TT comes back to the JobTracker with task reports of a job, after that job has been killed and retired. So, it is a timing issue.
I was able to replicate the problem as follows:
- Set the number of completed jobs per user in memory to 1 on the JT.
- Submit a job with long running tasks.
- On one of the TTs running tasks, send a SIGSTOP to the TT.
- Kill the job and submit enough other jobs from the same user, so this job is retired and gets out of the JT's page.
- Submit a new job
- Send SIGCONT to the TT which was stopped above.
At this point, in the JT log, we can see the above exception.
To fix this problem, we discussed two possible approaches.
- Check for null in the lookup result for the job, and ignore the tasks if the job is not found in the MemoryMatcher. IOW, the memory computed as 'reserved' for tasks would be less than the actual memory being used at that point on that TT (because the running task is not being accounted for). If any scheduling happens based on this number, this might oversubscribe memory on the TT. The behavior may be unpredictable in this case.
- The second option is to check for null, and if this scenario is found, do not schedule anything to the TT. The rationale here is that the memory state cannot be correctly determined and we are conservatively not assigning anything to the TT, until the state can be correctly determined. In this case, we could potentially waste a heartbeat. Note in general, in the heartbeat response, the running tasks will be killed.
Evaluating the two options, and also considering the rare case in which this could happen, we decided to take the conservative approach and favor consistency of state over utilization. Hence, the proposal is to do the second option.