Thanks Sanjay and Konstantin for looking at this one.
First the use case: putting the namenode in safemode causes existing applications to fail. This is a severe problem for us. If you are using hdfs for running map-reduce jobs, then putting the namenode is safemode means that tasks fail immediately. A reduce task that has been running for a long long time will fail and has to start all over again. If you are running hbase on hdfs, then hundreds of hbase region servers will die when the namenode goes into safemode.
@Sanjay: I cluster that runs hbase typically has very few files, less than 100K files. It takes a few seconds to run the savenamespace command. I can generalize: if a user is running hbase on hdfs, then it makes more sense to make savenamespace wait for a few seconds (via the read/write lock) rather than writing special case code in the hbase region servers to handle SafeModeException. As far as backward compatibility is concerned, i can add a "-f" option to indicate "do the savenamespace even if namenode is not in safemode", but still think that this is an option that everybody will use.
Can one of you explain why we always required savenamespace to have namenode is safemode? isn't it always better to stall the workload rather to fail the workload?
@Konstantin: can you pl explain the precise problem you have in mind? The rollFSImage() call acquires the FSNamesystem writelock, so it cannot race with saveNamespace. Moreover, saveNamespace truncates the edits log and removes edits.new while rollFSImage will exit out if it does not find edits.new.