Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4843

When using DefaultTaskController, JobLocalizer not thread safe

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.1
    • Fix Version/s: 1.2.0
    • Component/s: tasktracker
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In our cluster, some times job will failed due to below exception:
      2012-12-03 23:11:54,811 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201212031626_1115_r_000023_0:
      org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/$username/jobcache/job_201212031626_1115/job.xml in any of the configured local directories
      at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:424)
      at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
      at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1175)
      at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1058)
      at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2213)

      The root cause is JobLocalizer is not thread safe.
      In DefaultTaskController.initializeJob method:
      JobLocalizer localizer = new JobLocalizer((JobConf)getConf(), user, jobid);
      but in JobLocalizer, it just simply keep the reference of the conf.
      When two TaskLauncher threads(mapLauncher and reduceLauncher) try to initializeJob at same time, it will have two JobLocalizer, but only one conf instance.
      So some times ttConf.setStrings(JOB_LOCAL_CTXT, localDirs) will reset previous job's conf.
      Then it will cause the previous job's job.xml stored at another user's dir.

        Attachments

        1. mr-4843.patch
          3 kB
          Karthik Kambatla
        2. MAPREDUCE-4843-branch-1.1.patch
          0.9 kB
          yunjiong zhao

          Issue Links

            Activity

              People

              • Assignee:
                kasha Karthik Kambatla
                Reporter:
                zhaoyunjiong yunjiong zhao
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: