Details
-
New Feature
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
It used to be possible to get hbck's verdict of cluster status from the command line, especially useful for headless deployments, i.e. without requiring a browser with sufficient connectivity to load a UI, or scrape information out of raw HTML, or write regex to comb over log4j output. The hbck tool's output wasn't particularly convenient to parse but it was straightforward to extract the desired information with a handful of regular expressions.
HBCK2 has a different design philosophy than the old hbck, which is to serve as a collection of small and discrete recovery and repair functions, rather than attempt to be a universal repair tool. This makes a lot of sense and isn't the issue at hand. Unfortunately the old hbck's utility for reporting the current cluster health assessment has not been replaced either in whole or in part. Instead:
HBCK2 is for fixes. For listings of inconsistencies or blockages in the running cluster, you go elsewhere, to the logs and UI of the running cluster Master. Once an issue has been identified, you use the HBCK2 tool to ask the Master to effect fixes or to skip-over bad state. Asking the Master to make the fixes rather than try and effect the repair locally in a fix-it tool's context is another important difference between HBCK2 and hbck1.
Developing custom tooling to mine logs and scrape UI simply to gain a top level assessment of system health is unsatisfying. There should be a convenient means for querying the system if issues that rise to the level of inconsistency, in the hbck parlance, are believed to be present. It would be relatively simple to bring back the experience of invoking a command line tool to deliver a verdict. This could be added to the hbck2 tool itself but given that hbase-operator-tools is a separate project an intrinsic solution is desirable.
An option that immediately comes to mind is modification of the Master's hbck.jsp page to provide a JSON formatted output option if the HTTP Accept header asks for text/json. However, looking at the source of hbck.jsp, it makes more sense to leave it as is and implement a convenient machine parseable output format elsewhere. This can be trivially accomplished with a new servlet. Like hbck.jsp the servlet implementation would get a reference to HbckChore and present the information this class makes available via its various getters.
The machine parseable output is sufficient to enable headless hbck status checking but it still would be nice if we could provide operators a command line tool that formats the information for convenient viewing in a terminal. That part could be implemented in the hbck2 tool after this proposal is implemented.
