Major Description
When security is enabled on 0.22, MRASyncDiskService doesn't always delete the parent directories under toBeDeleted.
MRAsyncDiskService goes through toBeDeleted and creates "tasks" to delete the directories under there using the LinuxTaskController. It chooses which user to run as by looking at who owns that directory.
For example:
ls -al /mapred/local/toBeDeleted/2013-07-05_05-37-49.052_0 total 12 drwxr-xr-x 3 mapred mapred 4096 Jul 5 05:37 . drwxr-xr-x 5 mapred mapred 4096 Dec 19 10:15 .. drwxr-s--- 4 test mapred 4096 Jul 2 02:54 test
It would create a task to use "test" user to delete /mapred/local/toBeDeleted/2013-07-05_05-37-49.052_0/test (there could be more in there for other users). It then creates a task to use "mapred" user to delete /mapred/local/toBeDeleted/2013-07-05_05-37-49.052_0.
So, the problem is that we normally configure "mapred" to not be allowed by the LinuxTaskController in the /etc/hadoop/conf.cloudera.mapreduce1/taskcontroller.cfg. The permissions on the toBeDeleted dir is drwxr-xr-x mapred:mapred, which means that only "mapred" can delete things in it (i.e. the timestamped dirs). However, the MRAsyncDiskService is already running as the mapred user, so there's no reason to use the LinuxTaskController for impersonation anyway; we can directly do it from the Java code.
Another issue is that MRAsyncDiskService#deletePathsInSecureCluster expects an absolute file path (e.g. /mapred/local/toBeDeleted/2013-07-05_05-37-49.052_0, but MRAsyncDiskService#moveAndDeleteRelativePath passes in a relative path (e.g. toBeDeleted/2013-07-05_05-37-49.052_0).