[KUDU-1779] Consensus "stuck" with all transaction trackers are at limit

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.1.0
• Component/s: consensus
• Labels: stability
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 02/Dec/16 00:55
• Updated: 02/Jun/20 13:50

Description

In a stress cluster, I saw one tablet get "stuck" in the following state:

• the transaction_tracker on all three replicas is "full" (no more can be submitted)
• leader elections proceed just fine, but no leader is able to advance the commit index

The issue seems to be that a replica will respond with 'CANNOT_PREPARE' when its transaction tracker is full. The leader then ignores this response, and doesn't advance the majority-replicated watermark. The transaction tracker stays full forever because the in-flight transactions can't get committed.

Notes to follow.

Issue Links

• is related to: KUDU-1787 Tablet fails to start pending replicates due to transaction tracker limit