I suggested this in a ML thread but it seems that nobody actually tried it. I think it's worth following up on:
You could try setting the compaction thread to a lower priority. You could add a thread priority to NamedThreadPool, and pass that up from CompactionExecutor constructor. According to http://www.javamex.com/tutorials/threads/priority_what.shtml you have to run as root and add a JVM option to get this to work.
In particular, Brandon saw stress.py read latencies spike to 100ms during [anti]compaction on a 2 core machine. I'd like to see if this can mitigate that.