[KAFKA-15541] Improved StateStore Iterator metrics for detecting leaks

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Labels: kip kip-required
• Assignee: Nicholas Telford
• Reporter: Nicholas Telford
• Created: 04/Oct/23 14:04
• Updated: Yesterday 21:01

Description

KIP-989: RocksDB Iterator Metrics

RocksDB Iterators must be closed after use, to prevent memory leaks due to blocks being "pinned" in-memory. Pinned blocks can currently be tracked via the per-store block-cache-pinned-usage metric. However, it's common (and even recommended) to share the Block Cache among all stores in an application, to enable users to globally bound native memory used by RocksDB. This results in the block-cache-pinned-usage reporting the same memory usage for every store in the application, irrespective of which store is actually pinning blocks in the block cache.

To aid users in finding leaked Iterators, as well as identifying the cause of a high number of pinned blocks, we introduce two new metrics.

Issue Links

• duplicates: KAFKA-15437 Add metrics about open iterators
• links to: GitHub Pull Request #15975
• links to: GitHub Pull Request #16028
• links to: GitHub Pull Request #16041
• links to: GitHub Pull Request #16076
• mentioned in: Page Loading...