Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      cassandra: 3.11.3
      jre: openjdk version "1.8.0_181"
      heap size: 2GB
      memory limit: 3GB (cgroup)

      I started one of the nodes with "-Djdk.nio.maxCachedBufferSize=262144" but that did not seem to make any difference.

      Description

      While testing a 3 node 3.11.3 cluster I noticed that the nodes were suddenly killed by the Linux OOM killer after running without issues for 4-5 weeks.

      After enabling more metrics and leaving the nodes running for 12 days it sure looks like the
      "java.nio:type=BufferPool,name=direct" Mbean shows a very linear growth (approx 15MiB/24h, see attached screenshot). Is this expected to keep growing linearly after 12 days with a constant load?

       

      In my setup the growth/leak is about 15MiB/day so I guess in most setups it would take quite a few days until it becomes noticeable. I'm able to see the same type of slow growth in other production clusters even though the graph data is more noisy.

        Attachments

        1. cassandra.yaml
          56 kB
          Jonas Borgström
        2. CASSANDRA-15006-reference-chains.png
          384 kB
          Jonas Borgström
        3. cmdline.txt
          4 kB
          Jonas Borgström
        4. Screenshot_2019-02-04 Grafana - Cassandra.png
          422 kB
          Jonas Borgström
        5. Screenshot_2019-02-14 Grafana - Cassandra.png
          512 kB
          Jonas Borgström
        6. Screenshot_2019-02-14 Grafana - Cassandra(1).png
          276 kB
          Jonas Borgström
        7. Screenshot_2019-02-15 Grafana - Cassandra.png
          306 kB
          Jonas Borgström

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jborgstrom Jonas Borgström
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: