We observe that under hotspotting conditions that splitting will proceed very slowly and the "Cannot split region due to reference files being there" log line will be logged excessively. (branch-1 based production.) This is because after a region is split it must be compacted before it can be split again. Reference files must be replaced by real HFiles, normal housekeeping performed during compaction. However if the regionserver is under excessive load, its compaction queues may become deep. The daughters of a recently split hotspotting region may themselves continue to hotspot and will rapidly need to split again. If the scheduled compaction work to remove/replace reference files is queued hundreds or thousands of compaction queue elements behind current, the recently split daughter regions will not be able to split again for a long time and may grow very large, producing additional complications (very large regions, very deep replication queues).
To help avoid this condition we should prioritize the compaction of recently split daughter regions. Compaction requests include a priority field and CompactionRequest implements a comparator that sorts by this field. We already detect when a compaction request involves a region that has reference files, to ensure that it gets selected to be eligible for compaction, but we do not seem to prioritize the requests for post-split housekeeping. Split work should be placed at the top of the queue. Ensure that this is happening.