Description
In the event of a disk failure, the current workflow is to have operators:
- The Kudu operator shuts down Kudu for a maintenance window
- The data center operator replaces their disk
- The Kudu operator runs fs update_dirs
- The Kudu operator restarts Kudu
Step 3 is unlike what most systems do. As an operator, it would be nice to not have to do it. Once my disk is replaced, Kudu should just know that it's OK to start up (e.g. because it notices a completely empty disk where it expected an existing one), and perhaps run the update_dirs tool automatically.
An argument could be made that we shouldn't do this if we're not sure that the operator wants to, as replacing a disk may result in failed tablets. If the missing directory was caused by a simple user input error, maybe we shouldn't have run the tool and failed some tablets. But given many Kudu operators automate their deployment of Kudu, it's hard to think of a time when they wouldn't want to have Kudu run the tool.
In the case the tool fails because the "missing" directory ended up being a disk failure, we should simply start Kudu up with the data dir marked failed.
Attachments
Issue Links
- is cloned by
-
KUDU-2627 Automatically "fix" inconsistent data directories
-
- Resolved
-