Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2219

JT should not try to remove mapred.system.dir during startup

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the contents of mapred.system.dir instead of the directory itself.

      1. mapreduce-2219.txt
        5 kB
        Todd Lipcon
      2. mapreduce-2219.2.txt
        5 kB
        Todd Lipcon

        Activity

        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
        Hide
        tlipcon Todd Lipcon added a comment -

        test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.

        Show
        tlipcon Todd Lipcon added a comment - test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.
        Hide
        tomwhite Tom White added a comment -

        > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem.

        I see. The patch has tests for this code, so +1 (pending Hudson).

        Show
        tomwhite Tom White added a comment - > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. I see. The patch has tests for this code, so +1 (pending Hudson).
        Hide
        tlipcon Todd Lipcon added a comment -

        This patch fixes the permissions string in the comment, nice catch.

        Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.

        Show
        tlipcon Todd Lipcon added a comment - This patch fixes the permissions string in the comment, nice catch. Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.
        Hide
        tomwhite Tom White added a comment -

        This looks like a good change. A couple of comments:

        • Can you use FileUtil#fullyDeleteContents() rather than writing a new one?
        • The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Show
        tomwhite Tom White added a comment - This looks like a good change. A couple of comments: Can you use FileUtil#fullyDeleteContents() rather than writing a new one? The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Show
        tlipcon Todd Lipcon added a comment - https://reviews.apache.org/r/206/

          People

          • Assignee:
            tlipcon Todd Lipcon
            Reporter:
            tlipcon Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development