Devaraj, could you upmerge the patch. It no longer applies to trunk.
For the most part the patch looks OK to me. It seems to fix the issue at hand. However, I have been reading through the code and I find a few things a bit disturbing. I don't really have a problem with your fix, but I think if we refactored the code it would make things a lot cleaner. Because this is a blocker I am inclined to just check it in, on condition that we mark LocalDirAllocator.removeContext as deprecated and limited private for MapReduce and file a separate JIRA to clean things up. Also this is only an issue if the DefaultContainerExecutor is enabled, because when the LinuxContainerExecutor is used the problematic code runs in a separate process that releases everything when the localization is done.
The way the code in LocalDirAllocator is written there are really only two reasons for the static cache. First to avoid doing extra file system calls when creating the LocalDirAllocator. The second is to keep track of the last dir written to for round robin access. In the case of the App Cache this is never going to save any FS calls, because there is 1 and only 1 LocalDirAllocator created per application, and the key to the cache is different for each application. For the User cache this would save a bit, except for a bug in the code that makes the LocalDirAllocator cache key for the user cache separate for all applications (and making the leak much worse).
new LocalDirAllocator(String.format(USERCACHE_CTXT_FMT, appId));
conf.setStrings(String.format(USERCACHE_CTXT_FMT, appId), usersFileCacheDirs);
new LocalDirAllocator(String.format(USERCACHE_CTXT_FMT, user));
conf.setStrings(String.format(USERCACHE_CTXT_FMT, user), usersFileCacheDirs);
I would prefer to see the LocalDirAllocator cache have a maximum size, and have it remove entires in LRU order when it grows too big. This way we do not have to worry about removing entries we no longer need, and the worst thing that will happen is that if the cache is too small we will do some extra work, and potentially have disks written to in a more random order, instead of pure round robin.