I noticed that in the cases where the database was quite large a manual compaction of the database would shrink it from many gigabytes to under 100MB. It looks like we need periodic manual compactions of the database to keep the leveldb tables from getting filled up with stale keys. Once the database fills with mostly stale keys the recovery process becomes quite slow due to all the I/O required to iterate the few valid keys remaining.
Attaching a patch that adds a periodic full compaction of the database. By default it runs every hour, but the interval can be configured or even disabled if desired. I did some tests on a very large database writing keys every 10msec while a full compaction cycle was running, and the impact to the write performance was acceptable. Writes were occasionally delayed by up to 30% due to disk I/O contention, but overall the write performance was still quite good. If the database is already mostly compact the cycle runs very fast, so this should have minimal impact on the overall RM state store performance.