Hadoop Common
  1. Hadoop Common
  2. HADOOP-5641

Possible NPE in CapacityScheduler's MemoryMatcher

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MemoryMatches does a job lookup based on a JobID in its getMemReservedForTasks(TaskTrackerStatus taskTracker) method. By this time, job might have been removed from JT's memory resulting in a NPE.

      1. HADOOP-5641.patch
        8 kB
        Hemanth Yamijala

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -


        The particular method call in question is

         scheduler.taskTrackerManager.getJob(task.getTaskID().getJobID()).getJobConf(); 

        in MemoryMatcher.getMemReservedForTasks(TaskTrackerStatus taskTracker)

        Show
        Vinod Kumar Vavilapalli added a comment - The particular method call in question is scheduler.taskTrackerManager.getJob(task.getTaskID().getJobID()).getJobConf(); in MemoryMatcher.getMemReservedForTasks(TaskTrackerStatus taskTracker)
        Hide
        Hemanth Yamijala added a comment -

        The stack trace for the exception is as follows:

        2009-05-12 11:45:00,821 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 30300, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1a4036f, false, false, true, 292) from x.x.x.x:x error: java.io.IOException: java.lang.NullPointerException
        java.io.IOException: java.lang.NullPointerException
        at org.apache.hadoop.mapred.MemoryMatcher.getMemReservedForTasks(MemoryMatcher.java:144)
        at org.apache.hadoop.mapred.MemoryMatcher.matchesMemoryRequirements(MemoryMatcher.java:196)
        at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:407)
        at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:481)
        at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:277)
        at org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:895)
        at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2579)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)

        Show
        Hemanth Yamijala added a comment - The stack trace for the exception is as follows: 2009-05-12 11:45:00,821 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 30300, call heartbeat(org.apache.hadoop.mapred.TaskTrackerStatus@1a4036f, false, false, true, 292) from x.x.x.x:x error: java.io.IOException: java.lang.NullPointerException java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.mapred.MemoryMatcher.getMemReservedForTasks(MemoryMatcher.java:144) at org.apache.hadoop.mapred.MemoryMatcher.matchesMemoryRequirements(MemoryMatcher.java:196) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.getTaskFromQueue(CapacityTaskScheduler.java:407) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.assignTasks(CapacityTaskScheduler.java:481) at org.apache.hadoop.mapred.CapacityTaskScheduler$TaskSchedulingMgr.access$500(CapacityTaskScheduler.java:277) at org.apache.hadoop.mapred.CapacityTaskScheduler.assignTasks(CapacityTaskScheduler.java:895) at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2579) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:964) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:960) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:958)
        Hide
        Hemanth Yamijala added a comment -

        The reason for the above exception is as follows:

        • MemoryMatcher.getMemReservedForTasks tries to retrieve the JobConf for a job whose task has been reported in the TaskTrackerStatus in order to compute the available memory.
        • This job has been retired and hence the lookup returns null, which is not checked.
        • This scenario can occur if a TT comes back to the JobTracker with task reports of a job, after that job has been killed and retired. So, it is a timing issue.

        I was able to replicate the problem as follows:

        • Set the number of completed jobs per user in memory to 1 on the JT.
        • Submit a job with long running tasks.
        • On one of the TTs running tasks, send a SIGSTOP to the TT.
        • Kill the job and submit enough other jobs from the same user, so this job is retired and gets out of the JT's page.
        • Submit a new job
        • Send SIGCONT to the TT which was stopped above.

        At this point, in the JT log, we can see the above exception.

        To fix this problem, we discussed two possible approaches.

        • Check for null in the lookup result for the job, and ignore the tasks if the job is not found in the MemoryMatcher. IOW, the memory computed as 'reserved' for tasks would be less than the actual memory being used at that point on that TT (because the running task is not being accounted for). If any scheduling happens based on this number, this might oversubscribe memory on the TT. The behavior may be unpredictable in this case.
        • The second option is to check for null, and if this scenario is found, do not schedule anything to the TT. The rationale here is that the memory state cannot be correctly determined and we are conservatively not assigning anything to the TT, until the state can be correctly determined. In this case, we could potentially waste a heartbeat. Note in general, in the heartbeat response, the running tasks will be killed.

        Evaluating the two options, and also considering the rare case in which this could happen, we decided to take the conservative approach and favor consistency of state over utilization. Hence, the proposal is to do the second option.

        Show
        Hemanth Yamijala added a comment - The reason for the above exception is as follows: MemoryMatcher.getMemReservedForTasks tries to retrieve the JobConf for a job whose task has been reported in the TaskTrackerStatus in order to compute the available memory. This job has been retired and hence the lookup returns null, which is not checked. This scenario can occur if a TT comes back to the JobTracker with task reports of a job, after that job has been killed and retired. So, it is a timing issue. I was able to replicate the problem as follows: Set the number of completed jobs per user in memory to 1 on the JT. Submit a job with long running tasks. On one of the TTs running tasks, send a SIGSTOP to the TT. Kill the job and submit enough other jobs from the same user, so this job is retired and gets out of the JT's page. Submit a new job Send SIGCONT to the TT which was stopped above. At this point, in the JT log, we can see the above exception. To fix this problem, we discussed two possible approaches. Check for null in the lookup result for the job, and ignore the tasks if the job is not found in the MemoryMatcher. IOW, the memory computed as 'reserved' for tasks would be less than the actual memory being used at that point on that TT (because the running task is not being accounted for). If any scheduling happens based on this number, this might oversubscribe memory on the TT. The behavior may be unpredictable in this case. The second option is to check for null, and if this scenario is found, do not schedule anything to the TT. The rationale here is that the memory state cannot be correctly determined and we are conservatively not assigning anything to the TT, until the state can be correctly determined. In this case, we could potentially waste a heartbeat. Note in general, in the heartbeat response, the running tasks will be killed. Evaluating the two options, and also considering the rare case in which this could happen, we decided to take the conservative approach and favor consistency of state over utilization. Hence, the proposal is to do the second option.
        Hide
        Hemanth Yamijala added a comment -

        This patch fixes the NPE by implementing the proposal described in comments above. It checks for Null in the lookup for the job in MemoryMatcher, and returns nothing to the TT when such a state is determined. It also includes a test case for simulating this condition.

        Both the unit test and manual tests were run with and without the patch to make sure the fix works.

        Show
        Hemanth Yamijala added a comment - This patch fixes the NPE by implementing the proposal described in comments above. It checks for Null in the lookup for the job in MemoryMatcher, and returns nothing to the TT when such a state is determined. It also includes a test case for simulating this condition. Both the unit test and manual tests were run with and without the patch to make sure the fix works.
        Hide
        Hemanth Yamijala added a comment -

        Results of test-patch:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 3 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        As this patch touches only capacity scheduler tests, I ran all capacity scheduler tests and all passed.

        Show
        Hemanth Yamijala added a comment - Results of test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. As this patch touches only capacity scheduler tests, I ran all capacity scheduler tests and all passed.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        +1 for the patch.

        Show
        Vinod Kumar Vavilapalli added a comment - +1 for the patch.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.20

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.20
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/)
        . Fix a NullPointerException in capacity scheduler's memory based scheduling code when jobs get retired. Contributed by Hemanth Yamijala.

        Show
        Hudson added a comment - Integrated in Hadoop-trunk #834 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/ ) . Fix a NullPointerException in capacity scheduler's memory based scheduling code when jobs get retired. Contributed by Hemanth Yamijala.

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development