[HBASE-23836] Race condition in Master startup can ignore shutdown RPC

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha-1, 2.3.0
• Component/s: master
• Assignee: Unassigned
• Reporter: Nick Dimiduk
• Created: 13/Feb/20 18:41
• Updated: 13/May/20 23:33

Description

Identified as the root-cause of HBASE-23808. The master starts its RPC server first thing in its constructor, but does not initialize it's clusterTracker until later (both from the super constructor). Thus it is possible for a client to issue a shutdown RPC is received by a master before it has a clusterTracker instance. It is highly likely (guaranteed?) to also not be the active master at this point. In such a case, the shutdown is received and acknowledged back to the client, but is effectively ignored.

After chatting with bharathv on the PR, we believe this race becomes more likely (though still rare) after HBASE-18095. This is something to consider should we get around to a startup sequence refactor.

Setting priority as "minor" because the condition is merely annoying; from what I can tell, a subsequent shutdown RPC to any master would result in the correct shutdown of the first.

Issue Links

• is related to: HBASE-8422 Master won't go down. Stuck waiting on .META. to come on line.
• is related to: HBASE-18095 Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client
• is related to: HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBeforeStartingAnyRegionServer
• relates to: HBASE-24070 Master async shutdown rpc call may not complete