Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15006

Possible java.nio.DirectByteBuffer leak

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • cassandra: 3.11.3
      jre: openjdk version "1.8.0_181"
      heap size: 2GB
      memory limit: 3GB (cgroup)

      I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.

    • Normal

    Description

      While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.

      After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
      "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?

       

      In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.

      Attachments

        1. Screenshot_2019-02-25 Grafana - Cassandra.png
          448 kB
          Jonas Borgström
        2. Screenshot_2019-02-22 Grafana - Cassandra.png
          412 kB
          Jonas Borgström
        3. Screenshot_2019-02-15 Grafana - Cassandra.png
          306 kB
          Jonas Borgström
        4. cassandra.yaml
          56 kB
          Jonas Borgström
        5. cmdline.txt
          4 kB
          Jonas Borgström
        6. Screenshot_2019-02-14 Grafana - Cassandra(1).png
          276 kB
          Jonas Borgström
        7. Screenshot_2019-02-14 Grafana - Cassandra.png
          512 kB
          Jonas Borgström
        8. CASSANDRA-15006-reference-chains.png
          384 kB
          Jonas Borgström
        9. Screenshot_2019-02-04 Grafana - Cassandra.png
          422 kB
          Jonas Borgström

        Activity

          People

            Unassigned Unassigned
            jborgstrom Jonas Borgström
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: