Thanks a lot for the comments, Milind. Answers inline.
I think it is a "good thing" (tm) that NN makes HDFS readonly when nfs is not accessible.
I can see arguments for both. In fact, I originally argued in favor of the behavior you're describing. Upon further reflection, I think I've changed my opinion, however. At least, whatever policy is being used for the number of failed volumes that can be tolerated when syncing edit logs should also be used when checking for available resources in the NameNodeResourceChecker, for the purpose of consistency.
HDFS is getting public criticism about "losing" data, and if hdfs modifications are allowed by modifying a single destination, then it open up a window for losing data.
The purpose of configuring multiple dfs.name.dir directories is exactly so that the NN can tolerate multiple failures and keep on humming. It's not going to lose any data just because one goes offline - it will just write to the other directories.
The right thing to do is to return from safemode when the NFS volume becomes available again.
Please see this comment for the reasoning as to why the NameNodeResourceChecker doesn't automatically take the NN out of SM when it detects a volume being low on space.