Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-746

When a task tracker is killed, there is a Null Pointer exception thrown.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • None
    • None
    • jobtracker
    • None

    Description

      When a task tracker is killed, the job completes. But tehre is a null pointer exception thrown:

      java.io.IOException: java.lang.NullPointerException
      at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.removeHostCapacity(JobTracker.java:759)
      at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.blackListTracker(JobTracker.java:624)
      at org.apache.hadoop.mapred.JobTracker$FaultyTrackersInfo.incrementFaults(JobTracker.java:601)
      at org.apache.hadoop.mapred.JobTracker.finalizeJob(JobTracker.java:2337)
      at org.apache.hadoop.mapred.JobInProgress.garbageCollect(JobInProgress.java:2998)
      at org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2584)
      at org.apache.hadoop.mapred.JobInProgress.completedTask(JobInProgress.java:2473)
      at org.apache.hadoop.mapred.JobInProgress.updateTaskStatus(JobInProgress.java:1047)
      at org.apache.hadoop.mapred.JobTracker.updateTaskStatuses(JobTracker.java:3867)
      at org.apache.hadoop.mapred.JobTracker.processHeartbeat(JobTracker.java:3079)
      at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2817)
      at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)

      Steps to reproduce the issue:
      1) Bring up a 5 node cluster.
      2) set mapred.max.tracker.failures to 1
      3) Run a sleep command with 5 maps and 5 redcues.
      4) Kill a task tracker, when map is 80% complete.
      5) Kill the task tracker, by using "kill -9 process_id".
      6) At the time of killing, it was running 2 maps and 2 reduces.
      7) Allow 12 minutes to elapse for that task tracker to go out of job nodes list
      8) Then job completes successfully by giving the task attempts of the lost trackers to other nodes.
      9) The job tracker logs has this exception.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            iyappans Iyappan Srinivasan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment