Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1609

TaskTracker.localizeJob should not set permissions on job log directory recursively

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed TaskTracker so that it does not set permissions on job-log directory recursively. This fix both improves the performance of job localization as well as avoids a bug related to launching of task-cleanup attempts after TaskTracker's restart.

      Description

      Currently TaskTracker.localizeJob sets permissions (570 with LinuxTaskController) on job log directory recursively. When the tracker restarts/reinits, if there are new tasks for the job, localizeJob would find attempt log directories for the earlier tasks. If the job has many tasks, it would spend significant time in chmod'ing.

      Also, if a cleanup attempt is launched after the restart/reinit, we would hit MAPREDUCE-1607.

      This problem is missed by the patch for MAPREDUCE-927. The above problem never existed before MAPREDUCE:927.

      1. MAPREDUCE-1609-20-1.patch
        21 kB
        Hemanth Yamijala
      2. patch-1609.txt
        18 kB
        Amareshwari Sriramadasu
      3. patch-1609-1.txt
        18 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development