[KUDU-1078] Under heavy load, log cache reads return "Op in future"

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Private Beta
• Fix Version/s: 1.6.0
• Component/s: consensus
• Target Version/s: 1.6.0
• Code Review: http://gerrit.sjc.cloudera.com:8080/#/c/8038/
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 27/Aug/15 17:37
• Updated: 16/Feb/19 03:13
• Resolved: 20/Nov/17 19:51

Description

JD accidentally configured botl80 so that all nodes were writing to a single tablet (talk about a stress test!) I see the following warning occasionally in the leader logs:

W0827 11:34:04.051275 55950 consensus_queue.cc:272] T d6ff74fb04454712873abd0d2328e59b P 668314723deb4a818cc9a43eba51073c [LEADER]: The logs necessary to catch up peer 600e62435ca24425b29c00d9726b78be have been garbage collected. The follower will never be able to catch up (Not found: op in future). Instructing remote peer to remotely bootstrap.

Calling this a blocker because, if we couple this with the "automatically re-bootstrap nodes if they fall behind", we'll be accidentally deleting tablets when this happens (no good!)