Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15006

Possible java.nio.DirectByteBuffer leak



    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • cassandra: 3.11.3
      jre: openjdk version "1.8.0_181"
      heap size: 2GB
      memory limit: 3GB (cgroup)

      I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.

    • Normal


      While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.

      After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
      "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?


      In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.


        1. Screenshot_2019-02-25 Grafana - Cassandra.png
          448 kB
          Jonas Borgström
        2. Screenshot_2019-02-22 Grafana - Cassandra.png
          412 kB
          Jonas Borgström
        3. Screenshot_2019-02-15 Grafana - Cassandra.png
          306 kB
          Jonas Borgström
        4. Screenshot_2019-02-14 Grafana - Cassandra(1).png
          276 kB
          Jonas Borgström
        5. Screenshot_2019-02-14 Grafana - Cassandra.png
          512 kB
          Jonas Borgström
        6. Screenshot_2019-02-04 Grafana - Cassandra.png
          422 kB
          Jonas Borgström
        7. cmdline.txt
          4 kB
          Jonas Borgström
        8. CASSANDRA-15006-reference-chains.png
          384 kB
          Jonas Borgström
        9. cassandra.yaml
          56 kB
          Jonas Borgström



            Unassigned Unassigned
            jborgstrom Jonas Borgström
            0 Vote for this issue
            6 Start watching this issue