[KUDU-2135] Persist disk health state to disk

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs
• Assignee: Andrew Wong
• Reporter: Andrew Wong
• Created: 05/Sep/17 23:19
• Updated: 22/Feb/18 21:18

Description

When a tablet server disk fails, it is marked FAILED in memory and not touched during the lifetime of the tablet server. The next time the tablet server is started, however, if the disk happens to start up successfully, it will be used as is.

This may be risky, as the disk may be corrupted, or may be more prone to runtime failures. Additionally, when we begin striping metadata or WALs, we may end up with multiple of them for a single tablet (e.g. one that was on the failed disk, and another if the tablet was reassigned to the same tablet server). As such, the contents of the previously failed disk should not be used. It is thus necessary to persist the health of a disk to ensure unhealthy disks are not used.

Issue Links

• is related to: KUDU-2117 Spread metadata across multiple data directories