Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-807

Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: jobtracker
    • Labels:
      None
    • Release Note:
      Hide
      The JobTracker tries to delete the mapred.system.dir when it is starting up (with the job recovery disabled). The fix provided by this jira is that JobTracker will fail (bail out) with AccessControlException if it fails to delete files/directories in mapred.system.dir due to access control issues.
      Show
      The JobTracker tries to delete the mapred.system.dir when it is starting up (with the job recovery disabled). The fix provided by this jira is that JobTracker will fail (bail out) with AccessControlException if it fails to delete files/directories in mapred.system.dir due to access control issues.

      Description

      With restart disabled, the jobtracker does a rm -rf of the mapred.system.dir. If the mapred.system.dir contains user files with permissions other than 777 then the jobtracker gets stuck in a loop trying to delete the mapred.system.dir (and each time failing with AccessControlException). The JobTracker admin has to manually cleanup the mapred.system.dir if this happens.

      1. MAPRED-807-v1.1.patch
        0.7 kB
        Amar Kamat
      2. MAPRED-807-v1.2.patch
        0.7 kB
        Amar Kamat
      3. MAPRED-807-v1.3.patch
        2 kB
        Amar Kamat
      4. MAPRED-807-v1.4.patch
        2 kB
        Amar Kamat
      5. MAPRED-807-v1.6.patch
        13 kB
        Amar Kamat
      6. MAPREDUCE-807-v1.6-branch-0.20.patch
        7 kB
        Amar Kamat
      7. MAPRED-807-v1.7.patch
        13 kB
        Amar Kamat
      8. MAPREDUCE-807-v1.7-branch-0.20.patch
        7 kB
        Amar Kamat

        Activity

        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amar!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amar!
        Hide
        Amar Kamat added a comment -

        Attaching new patches after Devaraj's offline comments.

        Show
        Amar Kamat added a comment - Attaching new patches after Devaraj's offline comments.
        Hide
        Amar Kamat added a comment -

        Attaching a patch for branch 0.20.

        Show
        Amar Kamat added a comment - Attaching a patch for branch 0.20.
        Hide
        Amar Kamat added a comment -

        All ant tests passed except TestRecoveryManager and TestReduceFetch failed.

        Show
        Amar Kamat added a comment - All ant tests passed except TestRecoveryManager and TestReduceFetch failed.
        Hide
        Amar Kamat added a comment -

        Attaching a new patch for reivew thats doesnt mask LoginException and adds a testcase. Result of test-patch
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 19 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Running ant tests

        Show
        Amar Kamat added a comment - Attaching a new patch for reivew thats doesnt mask LoginException and adds a testcase. Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 19 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant tests
        Hide
        Amar Kamat added a comment -

        All tests except TestRecoveryManager FAILED (timeout) and TestReduceFetch FAILED (timeout) passed.

        Show
        Amar Kamat added a comment - All tests except TestRecoveryManager FAILED (timeout) and TestReduceFetch FAILED (timeout) passed.
        Hide
        Amar Kamat added a comment -

        Attaching a patch that fixes the issue at hand. Result of test-patch
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Running ant tests.

        Show
        Amar Kamat added a comment - Attaching a patch that fixes the issue at hand. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Running ant tests.
        Hide
        Amar Kamat added a comment -

        Result of test-patch
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        All tests (mapred+contrib) passed except TestRecoveryManager and TestReduceFetchFromPartialMem.

        Show
        Amar Kamat added a comment - Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. All tests (mapred+contrib) passed except TestRecoveryManager and TestReduceFetchFromPartialMem.
        Hide
        Amar Kamat added a comment -

        Attaching a patch. Testing in progress. Tested this patch manually and it works as expected.

        Show
        Amar Kamat added a comment - Attaching a patch. Testing in progress. Tested this patch manually and it works as expected.
        Hide
        Amar Kamat added a comment -

        Result of test-patch
        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 4 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        I wrote a testcase for this bug but unfortunately it was not the right way to do it and hence removed the testcase.

        All tests (mapred + contrib) passed except TestRecoveryManager (see MAPREDUCE-880).

        Show
        Amar Kamat added a comment - Result of test-patch [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 4 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. I wrote a testcase for this bug but unfortunately it was not the right way to do it and hence removed the testcase. All tests (mapred + contrib) passed except TestRecoveryManager (see MAPREDUCE-880 ).
        Hide
        Amar Kamat added a comment -

        Attaching a patch that guards the delete.

        Show
        Amar Kamat added a comment - Attaching a patch that guards the delete.
        Hide
        Amar Kamat added a comment -

        ant tests passed on my box.

        Show
        Amar Kamat added a comment - ant tests passed on my box.
        Hide
        Amar Kamat added a comment -

        Attaching a patch that solves the issue. Result of test-patch
        [exec] -1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
        [exec] Please justify why no new tests are needed for this patch.
        [exec] Also please list what manual steps were performed to verify this patch.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        Show
        Amar Kamat added a comment - Attaching a patch that solves the issue. Result of test-patch [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Amar Kamat
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development