Use absolute path for JobTracker's mapred.local.dir in MiniMRCluster

• Key: HADOOP-5828
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Hemanth Yamijala
• Reporter: Hemanth Yamijala
• Component/s: test
• Fix Version/s: 0.20.1

Description

MiniMRCluster's JobTracker sets the mapred.local.dir to a relative path: "build/test/mapred/local". While currently this does not cause any problem, if at some time in future, the JobTracker starts using LocalDirAllocator for localizing files, this could cause failures in contrib tests. This issue was faced when working with an internal patch that tried to introduce LocalDirAllocator into the JobTracker, and capacity scheduler test cases started failing.

Comments

[Hemanth Yamijala]

The failure was in DiskChecker.checkDir(), which called File.isDirectory() on the relative mapred.local.dir path. Debugging this, we found the issue to be one documented in http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4483097. When relative paths are passed to java.io.* APIs, some of them resolve them to absolute paths by prepending the path with the value of the system property user.dir. Some of them resolve them relative to the current working directory.

For contrib tests, we set the user.dir to a value that will typically be different from the current working directory. This triggers inconsistencies in the resolution to absolute files and causes APIs like File.isDirectory() to fail.

The recommendation from Sun is to use absolute rather than relative paths for such cases. Hence, the issue.

[Hemanth Yamijala]

This patch converts the relative path for the mapred.local.dir in MiniMRCluster to an absolute path.

[Sreekanth Ramakrishnan]

+1 to the patch. The changes looks fine.

[Hemanth Yamijala]

Results of test-patch:

[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]

[Hemanth Yamijala]

Running through Hudson.

[Hemanth Yamijala]

I ran tests locally. Got some failures, (like in streaming), but those tests are failing on trunk as well. Guess this is good to go.

[Hemanth Yamijala]

Patch for branch 0.20. Just changes the test directory structure.

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20.