[HBASE-16853] Regions are assigned to Region Servers in /hbase/draining after HBase Master failover

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 2.0.0
• Fix Version/s: 1.3.0, 0.98.24, 2.0.0
• Component/s: Balancer, Region Assignment
• Hadoop Flags: Reviewed
• Assignee: David Pope
• Reporter: David Pope
• Created: 16/Oct/16 10:11
• Updated: 09/Nov/17 02:48
• Resolved: 17/Oct/16 03:00

Description

Problem

If there are Region Servers registered as "draining", they will continue to have "draining" znodes after a HMaster failover; however, the balancer will assign regions to them.

How to reproduce (on hbase master):

1. Add regionserver to /hbase/draining: bin/hbase-jruby bin/draining_servers.rb add server1:16205
2. Unload the regionserver: bin/hbase-jruby bin/region_mover.rb unload server1:16205
3. Kill the Active HMaster and failover to the Backup HMaster
4. Run the balancer: hbase shell <<< "balancer"
5. Notice regions get assigned on new Active Master to Region Servers in /hbase/draining

Root Cause

The Backup HMaster initializes the DrainingServerTracker before the Region Servers are registered as "online" with the ServerManager. As a result, the ServerManager.drainingServers isn't populated with existing Region Servers in draining when we have an HMaster failover.

E.g.,

1. We have a region server in draining: server1,16205,1000
2. The RegionServerTracker starts up and adds a ZK watcher on the Znode for this RegionServer: /hbase/rs/server1,16205,1000
3. The DrainingServerTracker starts and processes each Znode under /hbase/draining, but the Region Server isn't registered as "online" so it isn't added to the ServerManager.drainingServers list.
4. The Region Server is added to the DrainingServerTracker.drainingServers list.
5. The Region Server's Znode watcher is triggered and the ZK watcher is restarted.
6. The Region Server is registered with ServerManager as "online".

END STATE: The Region Server has a Znode in /hbase/draining, but it is registered as "online" and the Balancer will start assigning regions to it.

$ bin/hbase-jruby bin/draining_servers.rb list
[1] server1,16205,1000

$ grep server1,16205,1000 logs/master-server1.log
2016-10-14 16:02:47,713 DEBUG [server1:16001.activeMasterManager] zookeeper.ZKUtil: master:16001-0x157c56adc810014, quorum=localhost:2181, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/server1,16205,1000

[2] 2016-10-14 16:02:47,722 DEBUG [server1:16001.activeMasterManager] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/server1,16205,1000

2016-10-14 16:02:47,730 DEBUG [server1:16001.activeMasterManager] zookeeper.ZKUtil: master:16001-0x157c56adc810014, quorum=localhost:2181, baseZNode=/hbase Set watcher on existing znode=/hbase/draining/server1,16205,1000

[3] 2016-10-14 16:02:47,731 WARN  [server1:16001.activeMasterManager] master.ServerManager: Server server1,16205,1000 is not currently online. Ignoring request to add it to draining list.

[4] 2016-10-14 16:02:47,731 INFO  [server1:16001.activeMasterManager] zookeeper.DrainingServerTracker: Draining RS node created, adding to list [server1,16205,1000]

2016-10-14 16:02:47,971 DEBUG [main-EventThread] zookeeper.ZKUtil: master:16001-0x157c56adc810014, quorum=localhost:2181, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev6918.prn2.facebook.com,16205,1476486047114

[5] 2016-10-14 16:02:47,976 DEBUG [main-EventThread] zookeeper.RegionServerTracker: Added tracking of RS /hbase/rs/server1,16205,1000

[6] 2016-10-14 16:02:52,084 INFO  [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16001] master.ServerManager: Registering server=server1,16205,1000