Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-15541

Improved StateStore Iterator metrics for detecting leaks

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.8.0
    • streams

    Description

      KIP-989: RocksDB Iterator Metrics

      RocksDB Iterators must be closed after use, to prevent memory leaks due to blocks being "pinned" in-memory. Pinned blocks can currently be tracked via the per-store block-cache-pinned-usage metric. However, it's common (and even recommended) to share the Block Cache among all stores in an application, to enable users to globally bound native memory used by RocksDB. This results in the block-cache-pinned-usage reporting the same memory usage for every store in the application, irrespective of which store is actually pinning blocks in the block cache.

      To aid users in finding leaked Iterators, as well as identifying the cause of a high number of pinned blocks, we introduce two new metrics.

      Attachments

        Issue Links

          Activity

            People

              nicktelford Nicholas Telford
              nicktelford Nicholas Telford
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: