We've recently started deploying 3.0.16 more heavily in production and today I noticed that full compaction of LCS tables takes a much longer time than it should. In particular it appears to be faster to convert a large dataset to STCS, run full compaction, and then convert it to LCS (with re-leveling) than it is to just run full compaction on LCS (with re-leveling).
I've attached the flame graph here which was generated by running Cassandra using -XX:+PreserveFramePointer, then using jstack to get the compaction native thread id (nid) which I then used perf to get on cpu time:
I took this data and collapsed it using the steps talked about in Brendan Gregg's java in flames blogpost (Instructions section) to generate the graph.
The results are that at least on this dataset (700GB of data compressed, 2.2TB uncompressed), we are spending 50% of our cpu time in moveStarts and I am unsure that we need to be doing that as frequently as we are. I'll see if I can come up with a clean reproduction to confirm if it's a general problem or just on this particular dataset.