Hadoop Common
  1. Hadoop Common
  2. HADOOP-5828

Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.

      1. HADOOP-5828.patch
        0.8 kB
        Hemanth Yamijala
      2. HADOOP-5828.20.patch
        0.7 kB
        Hemanth Yamijala

        Activity

        Hide
        Hemanth Yamijala added a comment -

        The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097. When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory.

        For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail.

        The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.

        Show
        Hemanth Yamijala added a comment - The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097 . When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory. For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail. The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.
        Hide
        Hemanth Yamijala added a comment -

        This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.

        Show
        Hemanth Yamijala added a comment - This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.
        Hide
        Sreekanth Ramakrishnan added a comment -

        +1 to the patch. The changes looks fine.

        Show
        Sreekanth Ramakrishnan added a comment - +1 to the patch. The changes looks fine.
        Hide
        Hemanth Yamijala added a comment -

        Results of test-patch:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 5 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]

        Show
        Hemanth Yamijala added a comment - Results of test-patch: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 5 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec]
        Hide
        Hemanth Yamijala added a comment -

        Running through Hudson.

        Show
        Hemanth Yamijala added a comment - Running through Hudson.
        Hide
        Hemanth Yamijala added a comment -

        I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.

        Show
        Hemanth Yamijala added a comment - I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.
        Hide
        Hemanth Yamijala added a comment -

        Patch for branch 0.20. Just changes the test directory structure.

        Show
        Hemanth Yamijala added a comment - Patch for branch 0.20. Just changes the test directory structure.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this to trunk and branch 0.20.

        Show
        Hemanth Yamijala added a comment - I just committed this to trunk and branch 0.20.

          People

          • Assignee:
            Hemanth Yamijala
            Reporter:
            Hemanth Yamijala
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development