Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-13929

BTree$Builder / io.netty.util.Recycler$Stack leaking memory

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.11.3
    • Component/s: Legacy/Core
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      Different to CASSANDRA-13754, there seems to be another memory leak in 3.11.0+ in BTree$Builder / io.netty.util.Recycler$Stack.

      • heap utilization increase after upgrading to 3.11.0 => cassandra_3.11.0_min_memory_utilization.jpg
      • No difference after upgrading to 3.11.1 (snapshot build) => cassandra_3.11.1_snapshot_heaputilization.png; thus most likely after fixing CASSANDRA-13754, more visible now
      • MAT shows io.netty.util.Recycler$Stack as top contributing class => cassandra_3.11.1_mat_dominator_classes.png
      • With -Xmx8G (CMS) and our load pattern, we have to do a rolling restart after ~ 72 hours

      Verified the following fix, namely explicitly unreferencing the recycleHandle member (making it non-final). In org.apache.cassandra.utils.btree.BTree.Builder.recycle()

              public void recycle()
              {
                  if (recycleHandle != null)
                  {
                      this.cleanup();
                      builderRecycler.recycle(this, recycleHandle);
                      recycleHandle = null; // ADDED
                  }
              }
      

      Patched a single node in our loadtest cluster with this change and after ~ 10 hours uptime, no sign of the previously offending class in MAT anymore => cassandra_3.11.1_mat_dominator_classes_FIXED.png

      Can' say if this has any other side effects etc., but I doubt.

        Attachments

        1. cassandra_3.11.0_min_memory_utilization.jpg
          287 kB
          Thomas Steinmaurer
        2. cassandra_3.11.1_mat_dominator_classes_FIXED.png
          17 kB
          Thomas Steinmaurer
        3. cassandra_3.11.1_mat_dominator_classes.png
          13 kB
          Thomas Steinmaurer
        4. cassandra_3.11.1_NORECYCLE_memory_utilization.jpg
          288 kB
          Thomas Steinmaurer
        5. cassandra_3.11.1_snapshot_heaputilization.png
          89 kB
          Thomas Steinmaurer
        6. cassandra_3.11.1_vs_3.11.2recyclernullingpatch.png
          74 kB
          Thomas Steinmaurer
        7. cassandra_heapcpu_memleak_patching_test_30d.png
          209 kB
          Thomas Steinmaurer
        8. dtest_example_80_request_fix.png
          631 kB
          Jay Zhuang
        9. dtest_example_80_request.png
          123 kB
          Jay Zhuang
        10. dtest_example_heap.png
          163 kB
          Jay Zhuang
        11. memleak_heapdump_recyclerstack.png
          140 kB
          Thomas Steinmaurer

        Issue Links

          Activity

            People

            • Assignee:
              jay.zhuang Jay Zhuang Assign to me
              Reporter:
              tsteinmaurer Thomas Steinmaurer
              Authors:
              Jay Zhuang
              Reviewers:
              Jason Brown

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment