[KUDU-1608] Catalog Manager DeleteTablet retry logic is broken

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: master
• Assignee: Dinesh Bhat
• Reporter: Dan Burkert
• Created: 13/Sep/16 01:40
• Updated: 18/Jan/17 01:08

Description

There are a couple of issues with the Catalog Manager's retry logic for DeleteTablet requests:

1. The retries loop indefinitely
2. The RPC response is checked against a whitelist of fatal errors, instead of a list of retriable errors. Additionally, we are missing many fatal errors on this list such as WRONG_SERVER_UUID and UNKNOWN_ERROR. I think we should instead only retry on errors which we know we can recover from.
3. The catalog manager aggressively sends out DeleteTablet requests to tablet servers when tablets are ejected from the group. Arguably this should only be done lazily when the dead tablets report in, since most of the time the tablet will be ejected due to failure (and will never be seen again).

Issue Links

• is related to: KUDU-1839 DNS failure during tablet creation lead to undeletable tablet