Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-4708

StorageProxy slow-down and memory leak

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.0.12, 1.1.6
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      I am consistently observing slow-downs in StorageProxy caused by the NonBlockingHashMap used indirectly by MessagingService via the callbacks ExpiringMap.

      This seems do be due to NBHM having unbounded memory usage in the face of workloads with high key churn. As monotonically increasing integers are used as callback id's by MessagingService, the backing NBHM eventually ends up growing the backing store unboundedly. This causes it to also do very large and expensive backing store reallocation and migrations, causing throughput to drop to tens of operations per second, lasting seconds or even minutes.

      This behavior is especially noticable for high throughput workloads where the dataset is completely in ram and I'm doing up to a hundred thousand reads per second.

      Replacing NBHM in ExpiringMap with the java standard library ConcurrentHashMap resolved the issue and allowed me to keep a consistent high throughput.

      An open issue on NBHM can be seen here: http://sourceforge.net/tracker/?func=detail&aid=3563980&group_id=194172&atid=948362

        Attachments

          Activity

            People

            • Assignee:
              danielnorberg Daniel Norberg
              Reporter:
              danielnorberg Daniel Norberg
              Authors:
              Daniel Norberg
              Reviewers:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: