Possible NPE in CapacityScheduler's MemoryMatcher

• Key: HADOOP-5641
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Hemanth Yamijala
• Reporter: Vinod K V
• Fix Version/s: 0.20.1

Description

MemoryMatches does a job lookup based on a JobID in its getMemReservedForTasks(TaskTrackerStatus taskTracker) method. By this time, job might have been removed from JT's memory resulting in a NPE.

Comments

[Vinod K V]

The particular method call in question is

scheduler.taskTrackerManager.getJob(task.getTaskID().getJobID()).getJobConf();

in MemoryMatcher.getMemReservedForTasks(TaskTrackerStatus taskTracker)

[Hemanth Yamijala]

The stack trace for the exception is as follows:

2009-05-12 11:45:00,821 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 30300, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1a4036f, false, false, true, 292) from x.x.x.x:x error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.mapred.MemoryMatcher.getMemReservedForTasks(MemoryMatcher.java:144)
at org.apache.hadoop.mapred.MemoryMatcher.matchesMemoryRequirements(MemoryMatcher.java:196)
at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:407)
at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:481)
at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:277)
at org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:895)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2579)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)

[Hemanth Yamijala]

The reason for the above exception is as follows:

• MemoryMatcher.getMemReservedForTasks tries to retrieve the JobConf for a job whose task has been reported in the TaskTrackerStatus in order to compute the available memory.
• This job has been retired and hence the lookup returns null, which is not checked.
• This scenario can occur if a TT comes back to the JobTracker with task reports of a job, after that job has been killed and retired. So, it is a timing issue.

I was able to replicate the problem as follows:

• Set the number of completed jobs per user in memory to 1 on the JT.
• Submit a job with long running tasks.
• On one of the TTs running tasks, send a SIGSTOP to the TT.
• Kill the job and submit enough other jobs from the same user, so this job is retired and gets out of the JT's page.
• Submit a new job
• Send SIGCONT to the TT which was stopped above.

At this point, in the JT log, we can see the above exception.

To fix this problem, we discussed two possible approaches.

• Check for null in the lookup result for the job, and ignore the tasks if the job is not found in the MemoryMatcher. IOW, the memory computed as 'reserved' for tasks would be less than the actual memory being used at that point on that TT (because the running task is not being accounted for). If any scheduling happens based on this number, this might oversubscribe memory on the TT. The behavior may be unpredictable in this case.

• The second option is to check for null, and if this scenario is found, do not schedule anything to the TT. The rationale here is that the memory state cannot be correctly determined and we are conservatively not assigning anything to the TT, until the state can be correctly determined. In this case, we could potentially waste a heartbeat. Note in general, in the heartbeat response, the running tasks will be killed.

Evaluating the two options, and also considering the rare case in which this could happen, we decided to take the conservative approach and favor consistency of state over utilization. Hence, the proposal is to do the second option.

[Hemanth Yamijala]

This patch fixes the NPE by implementing the proposal described in comments above. It checks for Null in the lookup for the job in MemoryMatcher, and returns nothing to the TT when such a state is determined. It also includes a test case for simulating this condition.

Both the unit test and manual tests were run with and without the patch to make sure the fix works.

[Hemanth Yamijala]

Results of test-patch:

[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

As this patch touches only capacity scheduler tests, I ran all capacity scheduler tests and all passed.

[Vinod K V]

+1 for the patch.

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20

[Hudson]

Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/)
. Fix a NullPointerException in capacity scheduler's memory based scheduling code when jobs get retired. Contributed by Hemanth Yamijala.