Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2219

JT should not try to remove mapred.system.dir during startup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During startup, the JT tries to clean up mapred.system.dir by recursively removing it and then recreating it. This requires that mapred.system.dir is inside a directory owned by the mapred user. For example, if set to /system/mapred then /system must be owned by the mapred account. This isn't documented properly and also seems unnecessary. Instead we can remove the contents of mapred.system.dir instead of the directory itself.

      1. mapreduce-2219.txt
        5 kB
        Todd Lipcon
      2. mapreduce-2219.2.txt
        5 kB
        Todd Lipcon

        Activity

        Todd Lipcon created issue -
        Todd Lipcon made changes -
        Field Original Value New Value
        Attachment mapreduce-2219.txt [ 12466984 ]
        Todd Lipcon made changes -
        Assignee Todd Lipcon [ tlipcon ]
        Todd Lipcon made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Show
        Todd Lipcon added a comment - https://reviews.apache.org/r/206/
        Hide
        Tom White added a comment -

        This looks like a good change. A couple of comments:

        • Can you use FileUtil#fullyDeleteContents() rather than writing a new one?
        • The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Show
        Tom White added a comment - This looks like a good change. A couple of comments: Can you use FileUtil#fullyDeleteContents() rather than writing a new one? The comment in the test for 755 perms should say "rwxr-xr-x" not "rwx-rx-rx".
        Hide
        Todd Lipcon added a comment -

        This patch fixes the permissions string in the comment, nice catch.

        Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.

        Show
        Todd Lipcon added a comment - This patch fixes the permissions string in the comment, nice catch. Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. Rather than filing a second JIRA to add that method to common, I just implemented it inline since it's pretty straightforward.
        Todd Lipcon made changes -
        Attachment mapreduce-2219.2.txt [ 12467608 ]
        Hide
        Tom White added a comment -

        > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem.

        I see. The patch has tests for this code, so +1 (pending Hudson).

        Show
        Tom White added a comment - > Unfortunately I couldn't use FileUtil.fullyDeleteContents because that method takes File and not Path/FileSystem. I see. The patch has tests for this code, so +1 (pending Hudson).
        Hide
        Todd Lipcon added a comment -

        test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.

        Show
        Todd Lipcon added a comment - test-patch and unit tests both passed (except for known timeouts). Will commit to trunk and 0.22 momentarily.
        Todd Lipcon made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #33 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/33/ )
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        11d 4h 24m 1 Todd Lipcon 27/Dec/10 02:36
        Patch Available Patch Available Resolved Resolved
        10d 23h 40m 1 Todd Lipcon 07/Jan/11 02:17
        Resolved Resolved Closed Closed
        339d 4h 2m 1 Konstantin Shvachko 12/Dec/11 06:19

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development