[HBASE-26245] Store region server list in master local region - ASF JIRA

Voters

Stop watching

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Brainstorming
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 2.5.0, 3.0.0-alpha-3
Component/s: master, Zookeeper
Labels:
None

Hadoop Flags:

Reviewed
Release Note:

Hide
A typical HBase deployment on cloud is to store the data other than WAL on OSS, and store the WAL data on a special HDFS cluster. A common operation is to rebuild the cluster with fresh new zk cluster and HDFS cluster, with only the old rootdir on OSS. But it requires extra manual steps since we rely on the WAL directory to find out previous live region servers, so we can schedule SCP to bring regions online.
After this issue, now it is possible to rebuild the cluster without extra manual steps as we will also store the previous live region servers in master local region.
But notice that you'd better stop masters first and then region servers when rebuilding, as some tests show that if there are some pending procedures, the new clusters may still hang.

Show
A typical HBase deployment on cloud is to store the data other than WAL on OSS, and store the WAL data on a special HDFS cluster. A common operation is to rebuild the cluster with fresh new zk cluster and HDFS cluster, with only the old rootdir on OSS. But it requires extra manual steps since we rely on the WAL directory to find out previous live region servers, so we can schedule SCP to bring regions online. After this issue, now it is possible to rebuild the cluster without extra manual steps as we will also store the previous live region servers in master local region. But notice that you'd better stop masters first and then region servers when rebuilding, as some tests show that if there are some pending procedures, the new clusters may still hang.

Description

Just a simple idea still need to be polished.

For large clusters, ZooKeeper could be a bottle neck, there are some related issues to not track region server list as much as possible, but what if we want to do more, just do not register region server on zk?

I think, zookeeper here is act as something like a service registry, we need the list to get all the region servers, and also need to know the changes of the list.
But in fact, we could also kill region server from master side, so the latter one could be done by a periodical heartbeat check daemon on master. And for the former one, we could store the list in master local region, so when master restart, it could setup the region server list by load from master local region.

And there are mainly two other side effect which are good:
1. We do not need to list the WAL directory on HDFS to find the previous region servers, for scheduling SCP. This could make it possible to restart a new HBase cluster based on only the root directory.
2. For now, region server needs to register to HMaster first, and then put its node on ZooKeeper, if it fails between these two actions, there is no way for HMaster to clean this dead server, as it never expires on ZooKeeper. There should be a related issue. If we just do not store a node on zk, then this problem is also gone.

Of course, there will still be lots of other problems, like whether we need another heartbeat call as reportForDuty is a bit heavy as we also report the region list, etc.