Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the hodring process. Problem is that it's not cleaned up at the end of the process; a use case would be:
- user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory is created
- user A deallocates the cluster, but that directory is not cleaned up
- user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring tries to write hdfs:///mapredsystem but it fails
- allocation succeeds, but there's no hodring running; looking at
0-jobtracker/logdir/hadoop.log under the temporary directory I can read:
2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException: Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
I guess a possible solution would be to clean up those directories during the deallocation process.