Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2219

JT should not try to remove mapred.system.dir during startup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the contents of mapred.system.dir instead of the directory itself.

      1. mapreduce-2219.2.txt
        5 kB
        Todd Lipcon
      2. mapreduce-2219.txt
        5 kB
        Todd Lipcon

        Activity

        Show
        Todd Lipcon added a comment - https://reviews.apache.org/r/206/
        Hide
        Tom White added a comment -

        This looks like a good change. A couple of comments:

        • Can you use FileUtil#fullyDeleteContents() rather than writing a new one?
        • The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Show
        Tom White added a comment - This looks like a good change. A couple of comments: Can you use FileUtil#fullyDeleteContents() rather than writing a new one? The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Hide
        Todd Lipcon added a comment -

        This patch fixes the permissions string in the comment, nice catch.

        Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.

        Show
        Todd Lipcon added a comment - This patch fixes the permissions string in the comment, nice catch. Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.
        Hide
        Tom White added a comment -

        > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem.

        I see. The patch has tests for this code, so +1 (pending Hudson).

        Show
        Tom White added a comment - > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. I see. The patch has tests for this code, so +1 (pending Hudson).
        Hide
        Todd Lipcon added a comment -

        test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.

        Show
        Todd Lipcon added a comment - test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development