[FLINK-23812] Support configuration of the RocksDB info logging via configuration

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.2
• Fix Version/s: 1.14.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available

Release Note

With RocksDB bumped to 6.20.3 (FLINK-14482), you can now also configure a rolling info logging strategy by configuring it accordingly via newly added state.backend.rocksdb.log.* settings. This can be helpful for debugging RocksDB (performance) issues in containerized environments where the local data dir is volatile but the logs should be retained on a separate volume mount.

• Assignee: Nico Kruber
• Reporter: Nico Kruber
• Created: 16/Aug/21 12:13
• Updated: 16/Aug/21 21:42
• Resolved: 16/Aug/21 21:39

Description

Since FLINK-14482 has been merged now, we should also allow users to configure more than just the log level (FLINK-20911) but also the following parameters so that they can safely enable RocksDB logging again (once disabled by default in FLINK-15068) by using a rolling logger, for example:

• max log file size via state.backend.rocksdb.log.max-file-size
• logging files to keep via state.backend.rocksdb.log.file-num
• log directory state.backend.rocksdb.log.dir, e.g. to put these logs onto a (separate) volume that may not be local and is retained after container shutdown for debugging purposes

Issue Links

• is blocked by: FLINK-14482 Bump up rocksdb version
• relates to: FLINK-20911 Support configuration of RocksDB log level
• relates to: FLINK-21336 Activate bloom filter in RocksDB State Backend via Flink configuration
• relates to: FLINK-15068 Disable RocksDB's local LOG by default
• links to: GitHub Pull Request #16848