Hadoop Common
  1. Hadoop Common
  2. HADOOP-4906

TaskTracker running out of memory after running several tasks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.1
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Fix the tasktracker for OOM exception by sharing the jobconf properties across tasks of the same job. Earlier a new instance was held for each task. With this fix, the job level configuration properties are shared across tasks of the same job.

      Description

      Looks like the TaskTracker isn't cleaning up correctly after completed/failed tasks, I suspect that the JobConfs aren't being deallocated. Eventually the TaskTracker runs out of memory after running several tasks.

      1. 4906_v5.patch
        2 kB
        Sharad Agarwal
      2. 4906_v4.patch
        1.0 kB
        Sharad Agarwal
      3. 4906_v3.patch
        4 kB
        Sharad Agarwal
      4. 4906_v2.patch
        4 kB
        Sharad Agarwal
      5. 4906_v1.patch
        1 kB
        Sharad Agarwal

        Activity

        Hide
        Arun C Murthy added a comment -

        I just committed this. Thanks, Sharad!

        Show
        Arun C Murthy added a comment - I just committed this. Thanks, Sharad!
        Hide
        Sharad Agarwal added a comment -

        incorporated Arun's comment

        Show
        Sharad Agarwal added a comment - incorporated Arun's comment
        Hide
        Arun C Murthy added a comment -

        Minor nit - we should remove TaskTracker.RunningJob.jobFile since it isn't used anymore...

        Show
        Arun C Murthy added a comment - Minor nit - we should remove TaskTracker.RunningJob.jobFile since it isn't used anymore...
        Hide
        Sharad Agarwal added a comment -

        same patch applies to 0.19 branch as well.

        Show
        Sharad Agarwal added a comment - same patch applies to 0.19 branch as well.
        Hide
        Sharad Agarwal added a comment -

        all tests passed. ant test-patch :
        -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no tests are needed for this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        It is not easy to write a test case for this.

        Show
        Sharad Agarwal added a comment - all tests passed. ant test-patch : -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no tests are needed for this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. It is not easy to write a test case for this.
        Hide
        Sharad Agarwal added a comment -

        does this bug exist in 0.19 as well? If so, can we get it into 0.19 branch too?

        yes. Will upload the patch for 0.19 shortly.

        Show
        Sharad Agarwal added a comment - does this bug exist in 0.19 as well? If so, can we get it into 0.19 branch too? yes. Will upload the patch for 0.19 shortly.
        Hide
        dhruba borthakur added a comment -

        does this bug exist in 0.19 as well? If so, can we get it into 0.19 branch too?

        Show
        dhruba borthakur added a comment - does this bug exist in 0.19 as well? If so, can we get it into 0.19 branch too?
        Hide
        Sharad Agarwal added a comment -

        Ran a SleepJob by putting a large dummy property value (20MB) in JobConf.

        • Without the patch, the OOM came after running 44 map tasks. This is expected as the default TT heap size is 1GB so the OOM should come within 50 tasks (20MB *50) executions.
        • With the patch, the no of tasks ran successfully past 1000, then I stopped it. During the run, monitored the heap size which remains almost steady.
        Show
        Sharad Agarwal added a comment - Ran a SleepJob by putting a large dummy property value (20MB) in JobConf. Without the patch, the OOM came after running 44 map tasks. This is expected as the default TT heap size is 1GB so the OOM should come within 50 tasks (20MB *50) executions. With the patch, the no of tasks ran successfully past 1000, then I stopped it. During the run, monitored the heap size which remains almost steady.
        Hide
        Arun C Murthy added a comment -

        This seems like a good interim solution. +1

        I'd be happy to commit this if we could run a large sleep-job and confirm the fix. Thanks!

        Show
        Arun C Murthy added a comment - This seems like a good interim solution. +1 I'd be happy to commit this if we could run a large sleep-job and confirm the fix. Thanks!
        Hide
        Sharad Agarwal added a comment -

        Being little clear on my last comment. The code could look like this in TaskTracker.java:

        private void localizeJob(TaskInProgress tip) throws IOException {
        .......
        .......
        synchronized (rjob) {
              if (!rjob.localized) {
                   ........
                   ........
                   //keep a reference to JobConf in RunningJob
                    rjob.localized = true;
                    rjob.jobConf = localJobConf;
             }
        }
            //create the task level JobConf 
            launchTaskForJob(tip, new JobConf(rjob.jobConf)); 
        }
        
        Show
        Sharad Agarwal added a comment - Being little clear on my last comment. The code could look like this in TaskTracker.java: private void localizeJob(TaskInProgress tip) throws IOException { ....... ....... synchronized (rjob) { if (!rjob.localized) { ........ ........ //keep a reference to JobConf in RunningJob rjob.localized = true ; rjob.jobConf = localJobConf; } } //create the task level JobConf launchTaskForJob(tip, new JobConf(rjob.jobConf)); }
        Hide
        Sharad Agarwal added a comment -

        It struck to me that there might be a better approach to the one we have taken, since the present one is hard to maintain.
        Currently there is a new JobConf object created for each task via new JobConf(rjob.jobFile) in TaskTracker#localizeJob. Since this object is created from a file, new references are created for all properties. Instead of that what if we create JobConf as below:

        //keep a reference to JobConf in RunningJob
        rjob.localized = true;
        rjob.jobConf = localJobConf;
        
        //create the task level JobConf
        launchTaskForJob(tip, new JobConf(rjob.jobConf)); 
        

        This way the properties for task's conf will be shallow cloned resulting in only reference copy instead of full new object creation. Also there would be a side benefit as there is no need to parse xml file and create JobConf for each task.
        Thoughts ?

        Show
        Sharad Agarwal added a comment - It struck to me that there might be a better approach to the one we have taken, since the present one is hard to maintain. Currently there is a new JobConf object created for each task via new JobConf(rjob.jobFile) in TaskTracker#localizeJob. Since this object is created from a file, new references are created for all properties. Instead of that what if we create JobConf as below: //keep a reference to JobConf in RunningJob rjob.localized = true ; rjob.jobConf = localJobConf; //create the task level JobConf launchTaskForJob(tip, new JobConf(rjob.jobConf)); This way the properties for task's conf will be shallow cloned resulting in only reference copy instead of full new object creation. Also there would be a side benefit as there is no need to parse xml file and create JobConf for each task. Thoughts ?
        Hide
        Sharad Agarwal added a comment -

        Setting the JobConf references to null in TaskInProgress#cleanup had problem since the TaskRunner thread may need the ref later. Had an offline discussion with Devaraj and decided that TaskRunner#run would be the better place to clean the references.
        This patch makes the JobConf references to null in the finally block of TaskRunner#run. Also had to keep some data in member variables of MapOutputFile and TaskInProgress so that JobConf reference is not required when task finishes.

        Show
        Sharad Agarwal added a comment - Setting the JobConf references to null in TaskInProgress#cleanup had problem since the TaskRunner thread may need the ref later. Had an offline discussion with Devaraj and decided that TaskRunner#run would be the better place to clean the references. This patch makes the JobConf references to null in the finally block of TaskRunner#run. Also had to keep some data in member variables of MapOutputFile and TaskInProgress so that JobConf reference is not required when task finishes.
        Hide
        dhruba borthakur added a comment -

        Since this problem affects the 0.19 branch, can we get this fix into the 0.19 branch as well? Thanks.

        Show
        dhruba borthakur added a comment - Since this problem affects the 0.19 branch, can we get this fix into the 0.19 branch as well? Thanks.
        Hide
        Sharad Agarwal added a comment -

        this patch sets the TaskInProgress#localJobConf object references to null.

        Show
        Sharad Agarwal added a comment - this patch sets the TaskInProgress#localJobConf object references to null.
        Hide
        Sharad Agarwal added a comment -

        sigh! just setting TaskInProgress.localJobConf to null doesn't seem to make JobConf object getting garbage collected. TaskInProgress.localJobConf references are being indirectly held by other classes -> MapOutputFile, MapTask and MapTaskRunner. so we need to set jobConf in these as well to null.

        Show
        Sharad Agarwal added a comment - sigh! just setting TaskInProgress.localJobConf to null doesn't seem to make JobConf object getting garbage collected. TaskInProgress.localJobConf references are being indirectly held by other classes -> MapOutputFile, MapTask and MapTaskRunner. so we need to set jobConf in these as well to null.
        Hide
        Devaraj Das added a comment -

        I guess we need to just set defaultJobConf/localJobConf to null in TaskInProgress.cleanup for now

        +1. I would say that we also nullify the TIP.diagnosticInfo since this is user settable too. Since a TIP has no other user settable data, and all TIPs of a particular job are removed when the latter completes, it should be okay to keep the TIPs in memory..

        Show
        Devaraj Das added a comment - I guess we need to just set defaultJobConf/localJobConf to null in TaskInProgress.cleanup for now +1. I would say that we also nullify the TIP.diagnosticInfo since this is user settable too. Since a TIP has no other user settable data, and all TIPs of a particular job are removed when the latter completes, it should be okay to keep the TIPs in memory..
        Hide
        Arun C Murthy added a comment -

        These are not really required to be hanging around till the job completion. As soon as the task finishes, these can be cleared.

        Unfortunately this isn't true. We need the TaskInProgress object in TaskTracker.tasks at least for failing the TaskAttempt of maps whose map-outputs are lost... see TaskTracker.mapOutputLost. I guess we need to just set defaultJobConf/localJobConf to null in TaskInProgress.cleanup for now? Sigh!


        Unrelated rant: My head still hurts from having to track this down in the mess that the TaskTracker has evolved into... I guess we need to seriously start thinking of cleaning up the TaskTracker, moving TaskTracker.TaskInProgress (very bad name too!) to it's own file, simplifying the interaction between the child TaskAttempt and the TaskTracker etc. Thoughts?

        Show
        Arun C Murthy added a comment - These are not really required to be hanging around till the job completion. As soon as the task finishes, these can be cleared. Unfortunately this isn't true. We need the TaskInProgress object in TaskTracker.tasks at least for failing the TaskAttempt of maps whose map-outputs are lost... see TaskTracker.mapOutputLost. I guess we need to just set defaultJobConf/localJobConf to null in TaskInProgress.cleanup for now? Sigh! Unrelated rant: My head still hurts from having to track this down in the mess that the TaskTracker has evolved into... I guess we need to seriously start thinking of cleaning up the TaskTracker, moving TaskTracker.TaskInProgress (very bad name too!) to it's own file, simplifying the interaction between the child TaskAttempt and the TaskTracker etc. Thoughts?
        Hide
        Sharad Agarwal added a comment -

        By reducing the TaskTracker heap size and running SleepJob with high number of map tasks I could reproduce the OOM.
        With the attached patch, TaskInProgress objects are being garbage collected and I am not seeing the OOM with the same settings.

        Show
        Sharad Agarwal added a comment - By reducing the TaskTracker heap size and running SleepJob with high number of map tasks I could reproduce the OOM. With the attached patch, TaskInProgress objects are being garbage collected and I am not seeing the OOM with the same settings.
        Hide
        Sharad Agarwal added a comment -

        TaskInProgress object references are held by Map<TaskAttemptID, TaskInProgress> tasks and Map<JobID, RunningJob> runningJobs. These are not cleared until the job is complete.
        These are not really required to be hanging around till the job completion. As soon as the task finishes, these can be cleared.

        Show
        Sharad Agarwal added a comment - TaskInProgress object references are held by Map<TaskAttemptID, TaskInProgress> tasks and Map<JobID, RunningJob> runningJobs . These are not cleared until the job is complete. These are not really required to be hanging around till the job completion. As soon as the task finishes, these can be cleared.
        Hide
        Devaraj Das added a comment -

        Yes this is a bug. References to completed/failed TaskInProgress objects from a datastructure (TaskTracker.tasks Map datastructure) are never cleared. The TaskInProgress object also has a JobConf reference...

        Show
        Devaraj Das added a comment - Yes this is a bug. References to completed/failed TaskInProgress objects from a datastructure (TaskTracker.tasks Map datastructure) are never cleared. The TaskInProgress object also has a JobConf reference...
        Hide
        Johan Oskarsson added a comment -

        I have seen this happen too. In one case the input path entry in our JobConf used up quite a bit of of ram since we were trying to merge a lot of small files into bigger ones. The JobConf for each task was kept in ram until the Job completed, but since we were running a lot of tasks the TaskTracker ran out of ram before the job could finish.

        Show
        Johan Oskarsson added a comment - I have seen this happen too. In one case the input path entry in our JobConf used up quite a bit of of ram since we were trying to merge a lot of small files into bigger ones. The JobConf for each task was kept in ram until the Job completed, but since we were running a lot of tasks the TaskTracker ran out of ram before the job could finish.
        Hide
        Arun C Murthy added a comment -

        A user complained about this on core-user@ supplying the following trace:

        java.lang.OutOfMemoryError: Java heap space 
        at java.util.Arrays.copyOf(Arrays.java:2882) 
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) 
        at java.lang.StringBuffer.append(StringBuffer.java:224) 
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1167)
        at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1120)
        at com.sun.org.apache.xerces.internal.dom.DeferredTextImpl.synchronizeData(DeferredTextImpl.java:93)
        at com.sun.org.apache.xerces.internal.dom.CharacterDataImpl.getData(CharacterDataImpl.java:160)
        at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:928)
        at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:851)
        at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:819) 
        at org.apache.hadoop.conf.Configuration.get(Configuration.java:278) 
        at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:446) 
        at org.apache.hadoop.mapred.JobConf.getKeepFailedTaskFiles(JobConf.java:308) 
        at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.setJobConf(TaskTracker.java:1506)
        at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:727)
        at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:721) 
        at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1306) 
        at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:946) 
        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1343) 
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2354)
        
        Show
        Arun C Murthy added a comment - A user complained about this on core-user@ supplying the following trace: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:224) at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1167) at com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1120) at com.sun.org.apache.xerces.internal.dom.DeferredTextImpl.synchronizeData(DeferredTextImpl.java:93) at com.sun.org.apache.xerces.internal.dom.CharacterDataImpl.getData(CharacterDataImpl.java:160) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:928) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:851) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:819) at org.apache.hadoop.conf.Configuration.get(Configuration.java:278) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:446) at org.apache.hadoop.mapred.JobConf.getKeepFailedTaskFiles(JobConf.java:308) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.setJobConf(TaskTracker.java:1506) at org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:727) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:721) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1306) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:946) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1343) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2354)

          People

          • Assignee:
            Sharad Agarwal
            Reporter:
            Arun C Murthy
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development