Hadoop Common
  1. Hadoop Common
  2. HADOOP-5828

Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.

      1. HADOOP-5828.patch
        0.8 kB
        Hemanth Yamijala
      2. HADOOP-5828.20.patch
        0.7 kB
        Hemanth Yamijala

        Activity

        Hemanth Yamijala created issue -
        Hide
        Hemanth Yamijala added a comment -

        The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097. When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory.

        For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail.

        The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.

        Show
        Hemanth Yamijala added a comment - The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097 . When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory. For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail. The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.
        Hide
        Hemanth Yamijala added a comment -

        This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.

        Show
        Hemanth Yamijala added a comment - This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.
        Hemanth Yamijala made changes -
        Field Original Value New Value
        Attachment HADOOP-5828.patch [ 12408105 ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        +1 to the patch. The changes looks fine.

        Show
        Sreekanth Ramakrishnan added a comment - +1 to the patch. The changes looks fine.
        Hide
        Hemanth Yamijala added a comment -

        Results of test-patch:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 5 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]

        Show
        Hemanth Yamijala added a comment - Results of test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]
        Hide
        Hemanth Yamijala added a comment -

        Running through Hudson.

        Show
        Hemanth Yamijala added a comment - Running through Hudson.
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hemanth Yamijala added a comment -

        I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.

        Show
        Hemanth Yamijala added a comment - I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.
        Hide
        Hemanth Yamijala added a comment -

        Patch for branch 0.20. Just changes the test directory structure.

        Show
        Hemanth Yamijala added a comment - Patch for branch 0.20. Just changes the test directory structure.
        Hemanth Yamijala made changes -
        Attachment HADOOP-5828.20.patch [ 12408236 ]
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.20.

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.20.
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.20.1 [ 12313866 ]
        Resolution Fixed [ 1 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        10h 1 Hemanth Yamijala 14/May/09 16:31
        Patch Available Patch Available Open Open
        17h 57m 1 Hemanth Yamijala 15/May/09 10:28
        Open Open Resolved Resolved
        17m 50s 1 Hemanth Yamijala 15/May/09 10:46

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development