[KUDU-1959] Hard to tell when a cluster is done starting up

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.16.0
• Component/s: ops-tooling
• Labels: roadmap-candidate usability
• Assignee: Abhishek Chennaka
• Reporter: Jean-Daniel Cryans
• Created: 28/Mar/17 21:28
• Updated: 03/Feb/22 21:59
• Resolved: 03/Feb/22 21:59

Description

Restarting a cluster that has a good amount of data, it's hard to tell when it's "done". Right now the things I do:

• Run ksck, wait until most tablets are not in "unavailable" or "boostrapping" state.
• Watch the metrics and see when the data under management is close to where it was before restarting (it grows as tablets are getting bootstrapped).
• Look at the tablet server web UIs for tablets, compare how many are done bootstrapping VS in the process of VS not started.

Ideas on how to improve this:

• In the master's web UI for tablet servers, show how many tablets are running VS not running (I wouldn't add anything about tombstoned tablets)
• Add metrics for tablets in different states.

Issue Links

• is duplicated by: KUDU-2424 Track cluster startup with ksck
• is related to: KUDU-3138 improve visibility into the bootstrapping status of a tablet server