Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Addressing reports of 'Unknown Server' has come up in various conversations lately. This issue is about fixing instances of 'Unknown Server' automatically as part of the tasks undertaken by CatalogJanitor when it runs.
First though, would like to figure a definition for 'Unknown Server' and a list of ways in which they arise. We need this to figure how to do safe auto-fixing.
Currently an 'Unknown Server' is a server found in hbase:meta that is not online (no recent heartbeat) and that is not mentioned in the dead servers list.
In outline, I'd think CatalogJanitor could schedule an expiration of the RS znode in zk (if exists) and then an SCP if it finds an 'Unknown Server'. Perhaps it waits for 2x or 10x the heartbeat interval just-in-case (or not). The SCP would clean up any references in hbase:meta by reassigning Regions assigned the 'Unknown Server' after replaying any WALs found in hdfs attributed to the dead server.
As to how they arise:
* A contrived illustration would be a large online cluster crashes down with a massive backlog of WAL files – zk went down for some reason say. The replay of the WALs look like it could take a very long time (lets say the cluster was badly configured and a bug and misconfig made it so each RS was carrying hundreds of WALs and there are hundreds of servers). To get the service back online, the procedure store and WALs are moved aside (for later replay with WALPlayer). The cluster comes up. meta is onlined but refers to server instances that are no longer around. Can schedule an SCP per server mentioned in the 'HBCK Report' by scraping and scripting hbck2 or, better, catalogjanitor could just do it.
* HBASE-24286 HMaster won't become healthy after after cloning... describes starting a cluster over data that is hfile-content only. In this case the original servers used manufacture the hfile cluster data are long dead yet meta still refers to the old servers. They will not make the 'dead servers' list.
Let this issue stew awhile. Meantime collect how 'Unknown Server' gets created and best way to fix.