[KUDU-1367] Crash at startup after full disk due to corrupt tablet metadata

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.5.0
• Component/s: tablet
• Target Version/s: 0.8.0
• Assignee: Unassigned
• Reporter: Mike Percy
• Created: 09/Mar/16 16:11
• Updated: 09/Mar/16 20:45
• Resolved: 09/Mar/16 20:45

Description

As reported by Nick Wolf.

The disk filled up on their tablet server, resulting in a bad metadata file being written. This was the error message on tablet server startup:

F0309 06:57:50.281460 32105 tablet_server_main.cc:49] Check failed: _s.ok() Bad status:Corruption: Failed to load FS layout: Could not read records from container
/var/lib/kudu/tserver/data/b1940ca83ce44da3bc6bf4a36f3d5e90:
Could not read body from proto container file
/var/lib/kudu/tserver/data/b1940ca83ce44da3bc6bf4a36f3d5e90.metadata:
File size not large enough to be valid: Proto container file
/var/lib/kudu/tserver/data/b1940ca83ce44da3bc6bf4a36f3d5e90.metadata:
tried to read /var/lib/kudu/tserver/data/b1940ca83ce44da3bc6bf4a36f3d5e90.metadata
bytes at offset 30 but file size is only 114688

Issue Links

• is duplicated by: KUDU-668 Log block container metadata files should be more forgiving to truncation