Attaching first-cut patch that adds unit test for MR-2413.
The unit test checks if good mapred local dirs of TaskTracker are getting updated or not when disks fail. To mimic disk failure, I replaced the specific mapred-local-directory with a regular-file of the same name — thus leading to DiskChecker.checkDir() to fail.

Please review the patch and provide your comments.

Eli, would you like to code review these?

Took a quick look:

In prepareDirToFail it says the file is set to perms "000" but File#createNewFile uses the default perms (eg 644 with umask 022), so it should still be accessible right?

If you want not always have waitForDiskHealthCheck wait for 10s at a time seems like you can lower the DISK_HEALTH_CHECK_INTERVAL to eg 1s.

Would also be good to test startup with a failed directory. Feel free to punt this to MAPREDUCE-2921.

Otherwise looks good.

>> In prepareDirToFail it says the file is set to perms "000" but File#createNewFile uses the default perms (eg 644 with umask 022), so it should still be accessible right?

No. The comment was wrong. I replace the directory with a file so that DiskChecker.checkDirs() will fail because it tries to do mkdir with the same name and this will be reported as a disk failure. Updating the comment accordingly.

>> If you want not always have waitForDiskHealthCheck wait for 10s at a time seems like you can lower the DISK_HEALTH_CHECK_INTERVAL to eg 1s.

OK. Changing to 1 sec.

>> Would also be good to test startup with a failed directory. Feel free to punt this to MAPREDUCE-2921.

This change would need a handle into the code of MiniMRCluster.TaskTrackerRunner() and needs some refactoring and exposing some api in MiniMRCluster. So not doing it as part of current JIRA.

Attaching new patch incorporating Eli's review comments.

+1 lgtm

Nit: you can remove the isDead=true in MiniMRCluster line 219 now that you unconditionally set it later.

Attaching new patch by removing the unnecessary line "isDead=true".

test-patch passed on my local machine.

+1

I saw ConcurrentModificationException in LocalStorage.checkDirs() in the logs of the new testcase because we are removing item from list while iterating over the list.

Attaching new patch here by fixing that code also as part of this JIRA.

Please review and provide your comments.

Good catch wrt CME. How about using an iterator instead? See attached, also removes some of the changes from another patch that were inadvertently added.

test-patch passed on my local machine.

I just committed this to branch-0.20-security. Thanks Eli.

Closed upon release of Hadoop-1.1.0.