[MAPREDUCE-746] When a task tracker is killed, there is a Null Pointer exception thrown.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Component/s: jobtracker
• Assignee: Unassigned
• Reporter: Iyappan Srinivasan
• Created: 10/Jul/09 08:11
• Updated: 06/Nov/09 06:20
• Resolved: 06/Nov/09 06:20

Description

When a task tracker is killed, the job completes. But tehre is a null pointer exception thrown:

java.io.IOException: java.lang.NullPointerException
at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.removeHostCapacity(JobTracker.java:759)
at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.blackListTracker(JobTracker.java:624)
at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.incrementFaults(JobTracker.java:601)
at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2337)
at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2998)
at org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2584)
at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2473)
at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1047)
at org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3867)
at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3079)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2817)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)

Steps to reproduce the issue:
1) Bring up a 5 node cluster.
2) set mapred.max.tracker.failures to 1
3) Run a sleep command with 5 maps and 5 redcues.
4) Kill a task tracker, when map is 80% complete.
5) Kill the task tracker, by using "kill -9 process_id".
6) At the time of killing, it was running 2 maps and 2 reduces.
7) Allow 12 minutes to elapse for that task tracker to go out of job nodes list
8) Then job completes successfully by giving the task attempts of the lost trackers to other nodes.
9) The job tracker logs has this exception.

Issue Links

• relates to: MAPREDUCE-754 NPE in expiry thread when a TT is lost