Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.12.7
-
None
-
None
Description
When setting config state.backend.rocksdb.memory.fixed-per-slot, all rocksdb instances in same slot are using shared WriteBufferManager.
I meet a extreme circumstances, there are 2 rocksdb using a WriteBufferManager size is 32M, if rocksdb-1 write (32*0.9)M data and won't have more data in a while, it won't trigger flush, then rocksdb-2 start writing data, it will trigger flush every single record and write many small file, spend a lot of time of backgroud compaction.
rocksdb-2 only flush current CF data, it won't recovery until rocksdb-1 flush data in memtable.
I can disable memory managed option to avoid this case, but I can't limit the memory usage.
Maybe can create a tracker monitor all rocksdb memory usage, trigger a force flush if it's necessary?
[1] RocksDB Write Buffer Manager https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager
Attachments
Issue Links
- relates to
-
FLINK-26050 Too many small sst files in rocksdb state backend when using time window created in ascending order
- Open