[KUDU-1073] Single TS falling too far behind hung YCSB

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Private Beta
• Fix Version/s: n/a
• Component/s: client, consensus
• Assignee: Jean-Daniel Cryans
• Reporter: Todd Lipcon
• Due: 18/Sep/15
• Created: 27/Aug/15 00:45
• Updated: 01/Apr/16 22:35
• Resolved: 01/Apr/16 22:35

Description

This caused a YCSB job to fail:

• a server fell behind for some reason (haven't done root cause on why – maybe just a bit slow)
• leader GCed the logs needed to catch it up, and thus stopped sending it any heartbeats or other messages
• the server had one write pending
• the java client apparently just kept retrying over and over against the same server

The server with the pending txn may actually have been the leader at the time it was written - otherwise not sure why Java keeps retrying it. Or perhaps the Java client got an error on the leader, failed over to try the follower, and RPCs to the follower are timing out.

Issue Links

• relates to: KUDU-952 [java client] RPCs that hit a busy server are duplicated
• relates to: KUDU-1003 Increase the default log retention