[CASSANDRA-14605] Major compaction of LCS tables very slow

• Type: Improvement
• Status: Open
• Priority: Low
• Resolution: Unresolved
• Component/s: Local/Compaction
• Labels: lcs performance

Environment

AWS, i3.4xlarge instance (very fast local nvme storage), Linux 4.13

Cassandra 3.0.16

• Assignee: Benedict Elliott Smith
• Reporter: Joey Lynch

Authors

Benedict Elliott Smith

• Created: 26/Jul/18 20:51
• Updated: 16/Apr/19 09:29

Description

We've recently started deploying 3.0.16 more heavily in production and today I noticed that full compaction of LCS tables takes a much longer time than it should. In particular it appears to be faster to convert a large dataset to STCS, run full compaction, and then convert it to LCS (with re-leveling) than it is to just run full compaction on LCS (with re-leveling).

I was able to get a CPU flame graph showing 50% of the major compaction's cpu time being spent in SSTableRewriter::maybeReopenEarly calling SSTableRewriter::moveStarts.

I've attached the flame graph here which was generated by running Cassandra using -XX:+PreserveFramePointer, then using jstack to get the compaction native thread id (nid) which I then used perf to get on cpu time:

perf record -t <compaction thread> -o <output file> -F 49 -g sleep 60 >/dev/null

I took this data and collapsed it using the steps talked about in Brendan Gregg's java in flames blogpost (Instructions section) to generate the graph.

The results are that at least on this dataset (700GB of data compressed, 2.2TB uncompressed), we are spending 50% of our cpu time in moveStarts and I am unsure that we need to be doing that as frequently as we are. I'll see if I can come up with a clean reproduction to confirm if it's a general problem or just on this particular dataset.

Issue Links

• relates to: CASSANDRA-14654 Reduce heap pressure during compactions