Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7546

AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.1.1
    • None
    • None
    • Normal

    Description

      In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition.

      Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets).

      Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545

      It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case.

      Attachments

        1. 7546.20_2.txt
          6 kB
          graham sanderson
        2. 7546.20_3.txt
          11 kB
          graham sanderson
        3. 7546.20_4.txt
          16 kB
          graham sanderson
        4. 7546.20_5.txt
          16 kB
          graham sanderson
        5. 7546.20_6.txt
          18 kB
          Benedict Elliott Smith
        6. 7546.20_7.txt
          18 kB
          graham sanderson
        7. 7546.20_7b.txt
          20 kB
          graham sanderson
        8. 7546.20_alt.txt
          16 kB
          graham sanderson
        9. 7546.20_async.txt
          22 kB
          graham sanderson
        10. 7546.20.txt
          4 kB
          Benedict Elliott Smith
        11. 7546.21_v1.txt
          12 kB
          graham sanderson
        12. cassandra-2.1-7546.txt
          10 kB
          graham sanderson
        13. cassandra-2.1-7546-v2.txt
          11 kB
          graham sanderson
        14. cassandra-2.1-7546-v3.txt
          12 kB
          graham sanderson
        15. graph2_7546.png
          245 kB
          graham sanderson
        16. graph3_7546.png
          253 kB
          graham sanderson
        17. graph4_7546.png
          232 kB
          graham sanderson
        18. graphs1.png
          300 kB
          graham sanderson
        19. hint_spikes.png
          25 kB
          graham sanderson
        20. suggestion1_21.txt
          26 kB
          graham sanderson
        21. suggestion1.txt
          30 kB
          graham sanderson
        22. young_gen_gc.png
          75 kB
          graham sanderson

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            graham.sanderson graham sanderson Assign to me
            graham sanderson graham sanderson
            graham sanderson
            Yuki Morishita
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment