[KUDU-3191] Fail tablet replicas that suffer from KUDU-2233 instead of crashing

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Component/s: compaction
• Assignee: Andrew Wong
• Reporter: Andrew Wong
• Created: 03/Sep/20 21:31
• Updated: 28/Sep/20 19:43
• Resolved: 28/Sep/20 19:43

Description

KUDU-2233 results in persisted corruption that causes a broken invariant, leading to a server crash. The recovery process for this corruption is arduous, especially if there are multiple tablet replicas in a given server that suffer from it – users typically start the server, see the crash, remove the affected replica manually via tooling, and restart, repeatedly until the server comes up healthily.

Instead, we should consider treating this as we do CFile block-level corruption[1] and fail the tablet replica. At best, we end up recovering from a non-corrupted replica. At worst, we'd end up with multiple corrupted replicas, which is still better than what we have today, which is multiple corrupted replicas and unavailable servers that lead to excessive re-replication.

[1] https://github.com/apache/kudu/commit/cf6927cb153f384afb649b664de1d4276bd6d83f

Issue Links

• is related to: KUDU-2793 Design a scan to detect KUDU-2233 corruption in a replica