[HBASE-18095] Provide an option for clients to find the server hosting META that does not involve the ZooKeeper client

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 1.7.0
• Component/s: Client
• Assignee: Bharath Vissapragada
• Reporter: Andrew Kyle Purtell
• Created: 23/May/17 17:17
• Updated: 03/Aug/21 19:59
• Resolved: 20/Feb/20 04:21

Description

Clients are required to connect to ZooKeeper to find the location of the regionserver hosting the meta table region. Site configuration provides the client a list of ZK quorum peers and the client uses an embedded ZK client to query meta location. Timeouts and retry behavior of this embedded ZK client are managed orthogonally to HBase layer settings and in some cases the ZK cannot manage what in theory the HBase client can, i.e. fail fast upon outage or network partition.

We should consider new configuration settings that provide a list of well-known master and backup master locations, and with this information the client can contact any of the master processes directly. Any master in either active or passive state will track meta location and respond to requests for it with its cached last known location. If this location is stale, the client can ask again with a flag set that requests the master refresh its location cache and return the up-to-date location. Every client interaction with the cluster thus uses only HBase RPC as transport, with appropriate settings applied to the connection. The configuration toggle that enables this alternative meta location lookup should be false by default.

This removes the requirement that HBase clients embed the ZK client and contact the ZK service directly at the beginning of the connection lifecycle. This has several benefits. ZK service need not be exposed to clients, and their potential abuse, yet no benefit ZK provides the HBase server cluster is compromised. Normalizing HBase client and ZK client timeout settings and retry behavior - in some cases, impossible, i.e. for fail-fast - is no longer necessary.

And, from ghelmling: There is an additional complication here for token-based authentication. When a delegation token is used for SASL authentication, the client uses the cluster ID obtained from Zookeeper to select the token identifier to use. So there would also need to be some Zookeeper-less, unauthenticated way to obtain the cluster ID as well.

Issue Links

• breaks: HBASE-23881 Netty SASL implementation does not wait for challenge response causing TestShadeSaslAuthenticationProvider failures
• breaks: HBASE-23889 Switch back to ZKConnectionRegistry by default at least in test
• is a parent of: PHOENIX-6523 Support for HBase Registry Implementations through Phoenix connection URL
• relates to: HBASE-23836 Race condition in Master startup can ignore shutdown RPC
• relates to: HBASE-23259 Ability to run mini cluster using pre-determined available random ports
• relates to: HBASE-23055 Alter hbase:meta
• links to: GitHub Pull Request #781
• links to: GitHub Pull Request #1167
• links to: GitHub Pull Request #1197
• links to: GitHub Pull Request #2280
• links to: Proposed design doc

Sub-Tasks

• Track ClusterID in stand by masters: Resolved
• Track active master server name in ActiveMasterManager: Resolved
• Track meta region changes on masters: Resolved
• Implement RPCs needed for master based registry: Resolved
• Master based registry implementation: Resolved
• Expose cluster ID for clients using it for delegation token based auth: Resolved
• Document HBASE-18095: Resolved
• Make MasterRegistry the default registry impl: Resolved
• Dynamic master discovery: Resolved