The suggested alternative compaction strategies don't sound very generally useful. We shouldn't maintain them in-tree.
Time bucketing and expiration (as implemented here) are very, very useful for timeseries data, and are in fact a blocker for production use of our timeseries systems. The requirement is that column families which store events at varying resolutions need to decay at different rates: there is no point keeping minute level resolution data indefinitely. Additionally, using TTLs is much, much too fine grained, and requires extra storage for each value.
We shouldn't have any provider-specific logic left in CompactionManager itself, it should all be based on the pluggable Strategy.
Agreed, but one approach that I think would be good middle ground would be to move doCompaction and doExpiration onto implementations of an AbstractCompactionTask, to be returned by the strategies that Alan has implemented. The 'task' concept already exists in this patch as an enum that CompactionManager switches on.
The interesting methods on CompactionStrategy are selectFor(Minor|Major)Compaction, which calculate the possible tasks to perform during a minor or major compaction. For the SizeTieredStrategy (aka, the strategy implemented in trunk), selectForMinor is the same as the previous getBuckets method.
The configuration changes in the patch are distracting, but it boils down to:
- Record the max client timestamp for sstables (useful for
- Allow for a compaction strategy to choose which files to perform a particular "task" on (bucketing)
- Implement a "task" for expiration (N files in, 0 files out)
- Add a strategy that uses the max client timestamp to expire old files