Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15006

Possible java.nio.DirectByteBuffer leak



    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • cassandra: 3.11.3
      jre: openjdk version "1.8.0_181"
      heap size: 2GB
      memory limit: 3GB (cgroup)

      I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.

    • Normal


      While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.

      After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
      "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?


      In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.


        1. cassandra.yaml
          56 kB
          Jonas Borgström
        2. CASSANDRA-15006-reference-chains.png
          384 kB
          Jonas Borgström
        3. cmdline.txt
          4 kB
          Jonas Borgström
        4. Screenshot_2019-02-04 Grafana - Cassandra.png
          422 kB
          Jonas Borgström
        5. Screenshot_2019-02-14 Grafana - Cassandra.png
          512 kB
          Jonas Borgström
        6. Screenshot_2019-02-14 Grafana - Cassandra(1).png
          276 kB
          Jonas Borgström
        7. Screenshot_2019-02-15 Grafana - Cassandra.png
          306 kB
          Jonas Borgström
        8. Screenshot_2019-02-22 Grafana - Cassandra.png
          412 kB
          Jonas Borgström
        9. Screenshot_2019-02-25 Grafana - Cassandra.png
          448 kB
          Jonas Borgström



            Unassigned Unassigned
            jborgstrom Jonas Borgström
            0 Vote for this issue
            6 Start watching this issue