[FLINK-7289] Memory allocation of RocksDB can be problematic in container environments

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 1.2.0, 1.3.0, 1.4.0, 1.7.2, 1.8.2, 1.9.0
• Fix Version/s: 1.10.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available

Release Note

After FLINK-7289, we could control the memory usage of RocksDB state backend. By default user could set the RocksDB memory boundary through `taskmanager.memory.managed.size` or `taskmanager.memory.managed.fraction`, tune the write/read memory ratio through `state.backend.rocksdb.memory.write-buffer-ratio` (by default 0.5) and the reserved memory fraction for index/filters through `state.backend.rocksdb.memory.high-prio-pool-ratio` (by default 0.1). We also supply a `state.backend.rocksdb.memory.fixed-per-slot` configuration for manually control, but for experts only. More details, please refer to the Flink documents.

• Assignee: Unassigned
• Reporter: Stefan Richter
• Created: 28/Jul/17 08:20
• Updated: 14/Aug/23 05:53
• Resolved: 04/Feb/20 03:48

Description

Flink's RocksDB based state backend allocates native memory. The amount of allocated memory by RocksDB is not under the control of Flink or the JVM and can (theoretically) grow without limits.
In container environments, this can be problematic because the process can exceed the memory budget of the container, and the process will get killed. Currently, there is no other option than trusting RocksDB to be well behaved and to follow its memory configurations. However, limiting RocksDB's memory usage is not as easy as setting a single limit parameter. The memory limit is determined by an interplay of several configuration parameters, which is almost impossible to get right for users. Even worse, multiple RocksDB instances can run inside the same process and make reasoning about the configuration also dependent on the Flink job.

Some information about the memory management in RocksDB can be found here:
https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB
https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

We should try to figure out ways to help users in one or more of the following ways:

• Some way to autotune or calculate the RocksDB configuration.
• Conservative default values.
• Additional documentation.

Issue Links

• is related to: FLINK-12785 RocksDB savepoint recovery can use a lot of unmanaged memory
• is related to: FLINK-32833 Rocksdb CacheIndexAndFilterBlocks must be true when using shared memory
• is related to: FLINK-15692 Enable limiting RocksDB memory consumption by default
• links to: GitHub Pull Request #10449

Sub-Tasks

• Build and supply frocksdb version to support WriteBufferManager: Closed
• Modify RocksDB backend to bound total memory via Cache with WriteBufferManager: Resolved
• Add documentation for memory control of RocksDB state backend with writebuffer manager: Closed
• Enable FRocksDB to optimize performance via options when bounding total memory: Resolved
• Make sure no resource leak of RocksObject: Closed
• Bump FRocksDB version for memory control: Closed
• Let MemoryManager allocate and track shared memory resources: Closed
• Add end-to-end test for controlling RocksDB memory usage: Closed
• Refactor the mechanism of how to constructure the cache and write buffer manager shared across RocksDB instances: Closed