Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
None
Description
Today when RocksDB is used as a State Store provider, memory usage when writing using writeBatch is not capped. Also, a related issue is that the state store coordinator can create multiple RocksDB instances on a single node without enforcing a global limit on native memory usage. Due to these issues we could run into OOM issues and task failures.
We are looking to improve this behavior by doing a series of improvements such as:
- remove writeBatch and use native RocksDB operations
- use writeBufferManager to manage global limit for all instances on a single node and accounting memtable + filter/index blocks usage as part of block cacheWith these changes we will be avoiding OOM issues around RocksDB native memory usage.