Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9805

nodetool status causes garbage to be accrued

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Tool/nodetool
    • Labels:
    • Environment:

      Ubuntu 14.04 64-bit
      Cassandra 2.0.14
      Java 1.7.0 OpenJDK

    • Severity:
      Normal

      Description

      As part of monitoring our Cassandra clusters (generally 2-6 nodes) we were running `nodetool status` regularly (~ every 5 minutes). On Cassandra 1.2.12 this worked fine and had negligible effect on the Cassandra database service.

      Having upgraded to Cassandra 2.0.14, we've found that, over time, the tenured memory space slowly fills with `RMIConnectionImpl` objects (and some other associated objects) until we start running into memory pressure and triggering proactive and then STW GC (which obviously impact performance of the cluster). It seems that these objects are kept around long enough to get promoted to tenured from Eden and then don't get considered for collection (due to internal reference cycles?).

      Very easy to reproduce, just call `nodetool status` in a loop and watch the memory usage climb to capacity then drop to empty after STW. No need to be accessing the DB keys at all.

        Attachments

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:

                Issue deployment