If there are Region Servers registered as "draining", they will continue to have "draining" znodes after a HMaster failover; however, the balancer will assign regions to them.
- Add regionserver to /hbase/draining: bin/hbase-jruby bin/draining_servers.rb add server1:16205
- Unload the regionserver: bin/hbase-jruby bin/region_mover.rb unload server1:16205
- Kill the Active HMaster and failover to the Backup HMaster
- Run the balancer: hbase shell <<< "balancer"
- Notice regions get assigned on new Active Master to Region Servers in /hbase/draining
The Backup HMaster initializes the DrainingServerTracker before the Region Servers are registered as "online" with the ServerManager. As a result, the ServerManager.drainingServers isn't populated with existing Region Servers in draining when we have an HMaster failover.
- We have a region server in draining: server1,16205,1000
- The RegionServerTracker starts up and adds a ZK watcher on the Znode for this RegionServer: /hbase/rs/server1,16205,1000
- The DrainingServerTracker starts and processes each Znode under /hbase/draining, but the Region Server isn't registered as "online" so it isn't added to the ServerManager.drainingServers list.
- The Region Server is added to the DrainingServerTracker.drainingServers list.
- The Region Server's Znode watcher is triggered and the ZK watcher is restarted.
- The Region Server is registered with ServerManager as "online".
END STATE: The Region Server has a Znode in /hbase/draining, but it is registered as "online" and the Balancer will start assigning regions to it.