Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7546

AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.1.1
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition.

      Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets).

      Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545

      It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case.

        Attachments

        1. suggestion1.txt
          30 kB
          graham sanderson
        2. suggestion1_21.txt
          26 kB
          graham sanderson
        3. 7546.20.txt
          4 kB
          Benedict Elliott Smith
        4. 7546.20_alt.txt
          16 kB
          graham sanderson
        5. 7546.20_2.txt
          6 kB
          graham sanderson
        6. 7546.20_3.txt
          11 kB
          graham sanderson
        7. 7546.20_4.txt
          16 kB
          graham sanderson
        8. 7546.20_5.txt
          16 kB
          graham sanderson
        9. 7546.20_6.txt
          18 kB
          Benedict Elliott Smith
        10. 7546.20_7.txt
          18 kB
          graham sanderson
        11. 7546.20_7b.txt
          20 kB
          graham sanderson
        12. young_gen_gc.png
          75 kB
          graham sanderson
        13. hint_spikes.png
          25 kB
          graham sanderson
        14. 7546.20_async.txt
          22 kB
          graham sanderson
        15. 7546.21_v1.txt
          12 kB
          graham sanderson
        16. graphs1.png
          300 kB
          graham sanderson
        17. graph2_7546.png
          245 kB
          graham sanderson
        18. graph3_7546.png
          253 kB
          graham sanderson
        19. cassandra-2.1-7546.txt
          10 kB
          graham sanderson
        20. cassandra-2.1-7546-v2.txt
          11 kB
          graham sanderson
        21. graph4_7546.png
          232 kB
          graham sanderson
        22. cassandra-2.1-7546-v3.txt
          12 kB
          graham sanderson

          Issue Links

            Activity

              People

              • Assignee:
                graham.sanderson graham sanderson
                Reporter:
                graham sanderson graham sanderson
                Authors:
                graham sanderson
                Reviewers:
                Yuki Morishita
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: