[KUDU-1954] Improve maintenance manager behavior in heavy write workload

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.3.0
• Component/s: compaction, perf, tserver
• Labels: performance roadmap-candidate scalability
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 22/Mar/17 17:42
• Updated: 28/Jul/21 06:29

Description

During the investigation in this doc I found a few maintenance-manager-related issues during heavy writes:

• we don't schedule flushes until we are already in "backpressure" realm, so we spent most of our time doing backpressure
• even if we configure N maintenance threads, we typically are only using ~50% of those threads due to the scheduling granularity
• when we do hit the "memory-pressure flush" threshold, all threads quickly switch to flushing, which then brings us far beneath the threshold
• long running compactions can temporarily starve flushes
• high volume of writes can starve compactions

Issue Links

• relates to: KUDU-1949 MM should trigger flushes before memory backpressure kicks in
• relates to: KUDU-2929 Don't starve compactions under memory pressure
• relates to: KUDU-3002 DMS may never flush when under memory pressure and there are on-going inserts