[KUDU-1587] Memory-based backpressure is insufficient on seek-bound workloads

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.10.0, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.7.1, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.11.1
• Fix Version/s: 1.13.0
• Component/s: tserver
• Labels: roadmap-candidate
• Code Review: http://gerrit.cloudera.org:8080/16343
• Assignee: Alexey Serbin
• Reporter: Todd Lipcon
• Created: 30/Aug/16 05:35
• Updated: 31/Aug/20 19:28
• Resolved: 31/Aug/20 19:28

Description

I pushed a uniform random insert workload from a bunch of clients to the point that the vast majority of bloom filters no longer fit in buffer cache, and the compaction had fallen way behind. Thus, every inserted row turns into 40+ seeks (due to non-compact data) and takes 400-500ms. In this kind of workload, the current backpressure (based on memory usage) is insufficient to prevent ridiculously long queues.

Issue Links

• is related to: KUDU-808 Throttle/reject writes to avoid running out of memory
• relates to: KUDU-2727 Contention on the Raft consensus lock can cause tablet service queue overflows