[KUDU-1768] Tablet server cannot recover after run out of disk space

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Labels: roadmap-candidate
• Assignee: Unassigned
• Reporter: Juan Yu
• Created: 29/Nov/16 18:10
• Updated: 28/May/20 15:37

Description

My node has small disk. after upsert into a large table several times, it ran out of disk space. I dropped the table. all nodes dropped the tablet successfully except one. it stuck in a bad state and tablet data are not removed even after restart.
For dropped table, it shows

tmp_store_sales	d88f8a7eba974cba8d9c547f69168c6b	hash buckets: (0)	FAILED (TABLET_DATA_DELETED): Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first			Invalid argument: Unable to delete on-disk data from tablet d88f8a7eba974cba8d9c547f69168c6b: The metadata for tablet d88f8a7eba974cba8d9c547f69168c6b still references orphaned blocks. Call DeleteTabletData() first

for other tables, it shows

kudu_store_sales	f539ce77657e431aa5ce4ab7f7375f84	hash buckets: (0, 7)	FAILED (TABLET_DATA_READY): IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)			IO error: Failed log replay. Reason: Failed to open new log: Insufficient disk space to allocate 67108864 bytes under path /dataroot/dataroot/tserver/wal/wals/f539ce77657e431aa5ce4ab7f7375f84/.tmp.newsegmentFdOSTp (30613504 bytes free vs 0 bytes reserved) (error 28)

attached tablet server log from this node.

Issue Links

• is related to: KUDU-320 Running out of space should be handled more gracefully