[KUDU-2404] Mitigate effects of full disks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs, tserver
• Labels: roadmap-candidate
• Assignee: Unassigned
• Reporter: Andrew Wong
• Created: 10/Apr/18 18:05
• Updated: 07/Jul/20 15:47

Description

Currently, if a tablet's data directory group runs out of space during a MRS or DMS flush, the operation will fail, and the tserver will crash, as MRS and DMS flush failures are fatal without the proper care. For disk failures, this "care" meant ensuring that upon failing the op, the tablet has started the process of shutting down and being failed so it can be replicated elsewhere. No such handling currently exists for full disks, although it wouldn't be unreasonable to apply the same or similar steps.

Issue Links

• is duplicated by: KUDU-761 TS seg fault after short log append
• is related to: KUDU-3114 tserver writes core dump when reporting 'out of space'
• is related to: KUDU-2405 Sanity-check tablet copies for full disks
• is related to: KUDU-2628 Allow hot-swapping of failed data directories
• is related to: KUDU-1172 Enable deleting tablets before or while they bootstrap
• is related to: KUDU-2795 Prevent cascading failures by detecting that disks are full and rejecting attempts to add additional replicas to a tablet server
• relates to: KUDU-2577 Support rebalancing data allocation across directories when adding a new data dir