Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1609

TaskTracker.localizeJob should not set permissions on job log directory recursively

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed TaskTracker so that it does not set permissions on job-log directory recursively. This fix both improves the performance of job localization as well as avoids a bug related to launching of task-cleanup attempts after TaskTracker's restart.

      Description

      Currently TaskTracker.localizeJob sets permissions (570 with LinuxTaskController) on job log directory recursively. When the tracker restarts/reinits, if there are new tasks for the job, localizeJob would find attempt log directories for the earlier tasks. If the job has many tasks, it would spend significant time in chmod'ing.

      Also, if a cleanup attempt is launched after the restart/reinit, we would hit MAPREDUCE-1607.

      This problem is missed by the patch for MAPREDUCE-927. The above problem never existed before MAPREDUCE:927.

      1. patch-1609-1.txt
        18 kB
        Amareshwari Sriramadasu
      2. patch-1609.txt
        18 kB
        Amareshwari Sriramadasu
      3. MAPREDUCE-1609-20-1.patch
        21 kB
        Hemanth Yamijala

        Issue Links

          Activity

          Hide
          Hemanth Yamijala added a comment -

          Attaching a patch on behalf of Amareshwari for earlier version of Hadoop. Not for commit here.

          Show
          Hemanth Yamijala added a comment - Attaching a patch on behalf of Amareshwari for earlier version of Hadoop. Not for commit here.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch for trunk.
          Patch incorporates MAPREDUCE-1595 as part of this fix, because the added testcase hits MAPREDUCE-1595 first and then this issue.

          Show
          Amareshwari Sriramadasu added a comment - Patch for trunk. Patch incorporates MAPREDUCE-1595 as part of this fix, because the added testcase hits MAPREDUCE-1595 first and then this issue.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12439432/patch-1609.txt
          against trunk revision 925561.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439432/patch-1609.txt against trunk revision 925561. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/45/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          All LinuxTaskController tests passed both as tt user and some other user.

          Show
          Amareshwari Sriramadasu added a comment - All LinuxTaskController tests passed both as tt user and some other user.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I started the review for this patch.

          Show
          Vinod Kumar Vavilapalli added a comment - I started the review for this patch.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch looks good. Only missed the javadoc for testTrackerReinit() and testTrackerRestart() from the backport patch. If you include those and Hudson blesses the current patch, this one will be good for commit.

          Show
          Vinod Kumar Vavilapalli added a comment - The patch looks good. Only missed the javadoc for testTrackerReinit() and testTrackerRestart() from the backport patch. If you include those and Hudson blesses the current patch, this one will be good for commit.
          Hide
          Amareshwari Sriramadasu added a comment -

          Patch adds javadoc for the newly added tests.

          Show
          Amareshwari Sriramadasu added a comment - Patch adds javadoc for the newly added tests.
          Hide
          Amareshwari Sriramadasu added a comment -

          All LinuxTaskController tests passed with the patch (both as tt user and some other user).

          Show
          Amareshwari Sriramadasu added a comment - All LinuxTaskController tests passed with the patch (both as tt user and some other user).
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the latest patch. Waiting for Hudson's blessings.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the latest patch. Waiting for Hudson's blessings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12442937/patch-1609-1.txt
          against trunk revision 938387.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442937/patch-1609-1.txt against trunk revision 938387. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/362/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The test-case failure of TestJobACL is because of MAPREDUCE-1727. Going to commit this.

          Show
          Vinod Kumar Vavilapalli added a comment - The test-case failure of TestJobACL is because of MAPREDUCE-1727 . Going to commit this.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this. Thanks Amareshwari!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this. Thanks Amareshwari!

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development