There is some issue with sstable metadata which is visible in system.log, the messages says:
WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading cardinality from Statistics.db failed for /opt/data/disk5/data/keyspace/table/mc-big-Data.db.
Although there is no such file.
The message has appeared after i've changed the compaction strategy from SizeTiered to Leveled. Compaction strategy has been changed region by region (total 3 regions) and it has coincided with the double client write traffic increase.
I have tried to run nodetool scrub to rebuilt the sstable, but that does not fix the issue.
So very hard to define the steps to reproduce, probably it will be:
- run stress tool with write traffic
- under load change compaction strategy from SireTiered to Leveled for the bunch of hosts
- add more write traffic
Reading the code it is said that if this metadata is broken, then "estimating the keys will be done using index summary".