[FLINK-18712] Flink RocksDB statebackend memory leak issue

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Information Provided
• Affects Version/s: 1.10.0
• Component/s: Runtime / State Backends
• Labels: usability
• Assignee: Yun Tang
• Reporter: Farnight
• Created: 25/Jul/20 02:49
• Updated: 30/Mar/21 12:49
• Resolved: 18/Nov/20 11:35

Description

When using RocksDB as our statebackend, we found it will lead to memory leak when restarting job (manually or in recovery case).

 

How to reproduce:

1. increase RocksDB blockcache size(e.g. 1G), it is easier to monitor and reproduce.
2. start a job using RocksDB statebackend.
3. when the RocksDB blockcache reachs maximum size, restart the job. and monitor the memory usage (k8s pod working set) of the TM.
4. go through step 2-3 few more times. and memory will keep raising.

 

Any solution or suggestion for this? Thanks!

Issue Links

• is related to: FLINK-19125 Avoid memory fragmentation when running flink docker image