Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1609

TaskTracker.localizeJob should not set permissions on job log directory recursively

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed TaskTracker so that it does not set permissions on job-log directory recursively. This fix both improves the performance of job localization as well as avoids a bug related to launching of task-cleanup attempts after TaskTracker's restart.

      Description

      Currently TaskTracker.localizeJob sets permissions (570 with LinuxTaskController) on job log directory recursively. When the tracker restarts/reinits, if there are new tasks for the job, localizeJob would find attempt log directories for the earlier tasks. If the job has many tasks, it would spend significant time in chmod'ing.

      Also, if a cleanup attempt is launched after the restart/reinit, we would hit MAPREDUCE-1607.

      This problem is missed by the patch for MAPREDUCE-927. The above problem never existed before MAPREDUCE:927.

      1. MAPREDUCE-1609-20-1.patch
        21 kB
        Hemanth Yamijala
      2. patch-1609.txt
        18 kB
        Amareshwari Sriramadasu
      3. patch-1609-1.txt
        18 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314045 ]
          Fix Version/s 0.22.0 [ 12314184 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Release Note Fixed TaskTracker so that it does not set permissions on job-log directory recursively. This fix both improves the performance of job localization as well as avoids a bug related to launching of task-cleanup attempts after TaskTracker's restart.
          Resolution Fixed [ 1 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1609-1.txt [ 12442937 ]
          Amareshwari Sriramadasu made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Summary TaskTracker.lcoalizeJob should not set permissions on job log directory recursively TaskTracker.localizeJob should not set permissions on job log directory recursively
          Description Currently TaskTracker.lcoalizeJob sets permissions (570 with LinuxTaskController) on job log directory recursively. When the tracker restarts/reinits, if there are new tasks for the job, localizeJob would find attempt log directories for the earlier tasks. If the job has many tasks, it would spend significant time in chmod'ing.

          Also, if a cleanup attempt is launched after the restart/reinit, we would hit MAPREDUCE-1607.
          Currently TaskTracker.localizeJob sets permissions (570 with LinuxTaskController) on job log directory recursively. When the tracker restarts/reinits, if there are new tasks for the job, localizeJob would find attempt log directories for the earlier tasks. If the job has many tasks, it would spend significant time in chmod'ing.

          Also, if a cleanup attempt is launched after the restart/reinit, we would hit MAPREDUCE-1607.

          This problem is missed by the patch for MAPREDUCE-927. The above problem never existed before MAPREDUCE:927.
          Amareshwari Sriramadasu made changes -
          Link This issue incorporates MAPREDUCE-1595 [ MAPREDUCE-1595 ]
          Amareshwari Sriramadasu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Amareshwari Sriramadasu made changes -
          Attachment patch-1609.txt [ 12439432 ]
          Hemanth Yamijala made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-1609-20-1.patch [ 12439278 ]
          Amareshwari Sriramadasu created issue -

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development