[HADOOP-242] job fails because of "No valid local directories in property: " exception

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3.2
• Assignee: Owen O'Malley
• Reporter: Yoram Arnon
• Created: 24/May/06 02:13
• Updated: 08/Jul/09 16:51
• Resolved: 23/Jun/06 23:39

Description

when running a fairly large job, of 70+K map tasks, I get many exceptions as shown below, and eventually the job failes when a task fails four times.
The exception doesn't really tell us enough information to debug this properly, so the first thing to do would be to add more information (path) to the exception.
The path indicated in the config file exists, is writable and valid, though 'path' may be anything.

the exception:
java.io.IOException: No valid local directories in property: mapred.local.dir at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:293) at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:153) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.localizeTask(TaskTracker.java:523) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.launchTask(TaskTracker.java:572) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:389) at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:303) at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:418) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:920)

the code:
public Path getLocalPath(String dirsProp, String path)
throws IOException {
String[] dirs = getStrings(dirsProp);
int hashCode = path.hashCode();
FileSystem fs = FileSystem.getNamed("local", this);
for (int i = 0; i < dirs.length; i++) { // try each local dir
int index = (hashCode+i & Integer.MAX_VALUE) % dirs.length;
Path file = new Path(dirs[index], path);
Path dir = file.getParent();
if (fs.exists(dir) || fs.mkdirs(dir))

{ return file; }

}
throw new IOException("No valid local directories in property: "+dirsProp);
}

Issue Links

• is related to: HADOOP-277 Race condition in Configuration.getLocalPath()