[CASSANDRA-7546] AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.1
• Severity: Normal
• Assignee: graham sanderson
• Reporter: graham sanderson

Authors

graham sanderson

Reviewers

Yuki Morishita

• Created: 15/Jul/14 00:56
• Updated: 16/Apr/19 09:31
• Resolved: 13/Oct/14 17:14

Description

In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition.

Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets).

Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545

It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case.

Issue Links

• is duplicated by: CASSANDRA-6534 Slow inserts with collections into a single partition (Pathological GC behavior)
• is related to: CASSANDRA-7545 Hints for a down node are written to a single partition in system.hints on the coordinator leading to contention