Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15006

Possible java.nio.DirectByteBuffer leak

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:

      cassandra: 3.11.3
      jre: openjdk version "1.8.0_181"
      heap size: 2GB
      memory limit: 3GB (cgroup)

      I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.

    • Severity:


      While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.

      After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
      "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?


      In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.


        1. cassandra.yaml
          56 kB
          Jonas Borgström
        2. CASSANDRA-15006-reference-chains.png
          384 kB
          Jonas Borgström
        3. cmdline.txt
          4 kB
          Jonas Borgström
        4. Screenshot_2019-02-04 Grafana - Cassandra.png
          422 kB
          Jonas Borgström
        5. Screenshot_2019-02-14 Grafana - Cassandra.png
          512 kB
          Jonas Borgström
        6. Screenshot_2019-02-14 Grafana - Cassandra(1).png
          276 kB
          Jonas Borgström
        7. Screenshot_2019-02-15 Grafana - Cassandra.png
          306 kB
          Jonas Borgström
        8. Screenshot_2019-02-22 Grafana - Cassandra.png
          412 kB
          Jonas Borgström
        9. Screenshot_2019-02-25 Grafana - Cassandra.png
          448 kB
          Jonas Borgström



            • Assignee:
              Unassigned Assign to me
              jborgstrom Jonas Borgström


              • Created:

                Issue deployment