Details
-
Improvement
-
Status: Resolved
-
Normal
-
Resolution: Not A Problem
-
None
-
None
-
None
Description
I'm questioning the design decision to use sstables for storing Hinted Handoffs.
sstables are optimized for reads. Things like bloom-filters, indexes, and the like - none of these are necessary for Hinted Handoffs.
After turning off Hinted Handoffs, I'm still seeing Hinted Handoffs be compacted a week later. The fact that they are compacted in the first place doesn't seem right. The whole purpose of compaction is to optimize sstables for reads, which doesn't apply here.
In our case, this is exacerbated by using Leveled Compaction. The overhead of compactions is significantly larger with LCS. When compactions begin to backup under heavy write load, Hinted Handoffs contribute to that.
Another thing that makes it worse (I think) is that Hinted Handoffs are stored on Coordinator nodes after 1.0. That means that these are being compacted across all key-ranges.
It seems that Hinted Handoffs should be persisted in a simple queue-like data-structure, that's not sorted by keys. The only thing the data-structure needs to support is the ability to replay them in order.
A simpler improvement would be to introduce max_hint_window_size_in_mb property, in addition to max_hint_window_in_ms. That would at least allow you to control how much these build up.