Until HDFS comes up with a solution to eliminate NN SPoF, old fashioned HA measures are required to keep NameNode available.
So far, best and seemingly reliable bet on Linux is to have network replicated block device, a heart beat providing messaging connection between HA nodes and a cluster resource manager software to keep track of infrastructural resource dependencies and moving them between machines in the requirement order.
All in all, HBase's tolerance window for NN unavailability mostly depends on particular load at the time of failover and RSs requirements to create new files.
Failing over to another node where a healthy replica of NN store exists and starting an NN instance will cause the NN to collect block information from every "new" and "unknown" DataNode for the first time. Additionally, default value for extension of safe mode after threshold reach is 30 seconds. (property: dfs.namenode.safemode.extension). This prolonged unavailability window can/will have bad effects on RSs. (jdcryans will comment his observations).
We implemented a NameNode HA cluster with open source tools like OpenAIS, Pacemaker, Heartbeat and DRBD.
- NameNode disk storage is replicated between two machines (adding a 3rd machine is possible with new DRBD).
- OpenAIS provides intra-cluster messaging and heart beat availability layer.
- Pacemaker is used to manage Cluster Resources. (DRBD disks, filesystem mount, NN service IP, NN daemon)
- An OCF script to start, stop, validate and monitor (periodic calls) the subsystem (NN, JT, SNN).
At the end of the day, this is applicable to not only NameNode but also to JobTracker and SecondaryNameNode.
For a starting point, ClusterLabs (creators of Pacemaker) offer an e-book on creating clusters with DRBD, Pacemaker, OpenAIS called Clusters From Scratch (PDF Link)