Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Later
-
None
-
None
-
None
-
None
Description
An open question is about how HBase would track these LogService-backed WALs.
Presently, HBase uses server-names and a well-known directory in HDFS to know what WALs exist. Since we are not relying on HDFS (or a distributed filesystem), we need to come up with something else.
sergey soldatov made a good suggestion today which was that we could implement another Ratis StateMachine whose purpose was specifically designed to managing the state of LogStreams "in HBase". This information should be relatively "small" (WRT the amount of data in each LogStream), so we can avoid the kinds of problems described in HBASE-20961 around re-introducing a failed peer to the quorum. This is the best idea I've heard so far on the matter.
The other obvious candidate would be ZooKeeper but this is probably a non-starter as it would be persistent data (which is an HBase anti-pattern).