[KUDU-2202] Removing a data directory is unsafe

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.7.0
• Component/s: fs
• Assignee: Adar Dembo
• Reporter: Adar Dembo
• Created: 27/Oct/17 02:43
• Updated: 22/Jan/18 05:40
• Resolved: 22/Jan/18 05:40

Description

I wrote a patch that modifies the Kudu CLI to allow for data directory addition and removal. It turns out that implementing removal safely is quite complicated. Below I've outlined the various issues and their potential solutions or workarounds:

1. The data dir to be removed may be the first data dir, used for tablet and consensus metadata. Until these can be striped to other data dirs, we'll work around this by prohibiting the removal of the first data dir outright.
2. Tablets may have data blocks on the removed directory. No problem, just consider those tablets to be failed. Except, this could lead to the failed tablets' block IDs being reused in the creation of new blocks, which can lead to all sorts of issues. For example, deletion of a failed tablet whose block IDs were reused means deleting another tablet's data. Some band-aid solutions here include rewriting the superblocks on removal to also strip out all block IDs that were on the removed directory or persisting the maximum block ID on every disk (or in every superblock) to prevent block ID reuse. More solutions are discussed in the aforementioned patch as well.
3. Even if the removed data dir is empty of data, existing tablets may still be configured to stripe to it, either explicitly (their data dir group includes this data dir) or implicitly (they're from an older version of Kudu and don't have a data dir group). We could work around this by rewriting these tablets' superblocks to prune the removed data dir from the tablets' data dir groups.

The patch works around issue #1 via prohibition, but is vulnerable to issues #2 and #3. Given that, and given that data dir addition is of limited value (we lack intra-node data rebalancing and existing tablets' superblocks aren't rewritten to take advantage of a new data dir), I'm shelving the patch. Note that it depends on this other patch, which allows tablets with removed data directories to start up.

Issue Links

• relates to: KUDU-2062 Tool to add and configure a data dir to an existing server