[FLINK-15068] Disable RocksDB's local LOG by default

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.2, 1.8.2, 1.9.1
• Fix Version/s: 1.10.0
• Component/s: Runtime / State Backends
• Labels: pull-request-available

Release Note

Logging in RocksDB (e.g., logging related to flush, compaction, memtable creation, etc.) has been disabled by default to prevent disk space from being filled up unexpectedly. Users that need to enable logging should implement their own RocksDBOptionsFactory that creates DBOptions instances with InfoLogLevel set to INFO_LEVEL.

• Assignee: Nico Kruber
• Reporter: Nico Kruber
• Created: 05/Dec/19 08:25
• Updated: 16/Aug/21 12:15
• Resolved: 07/Dec/19 15:37

Description

With Flink's default settings for RocksDB, it will write a log file (not the WAL, but pure logging statements) into the data folder. Besides periodic statistics, it will log compaction attempts, new memtable creations, flushes, etc.

A few things to note about this practice:

1. this LOG file is growing over time with no limit
2. the default logging level is INFO
3. the statistics in there may help looking into performance and/or disk space problems (but maybe you should be looking and monitoring metrics instead)
4. this file is not useful for debugging errors since it will be deleted along with the local dir when the TM goes down

With a custom OptionsFactory, the user can change the behaviour like the following:

    @Override
    public DBOptions createDBOptions(DBOptions currentOptions) {
        currentOptions = super.createDBOptions(currentOptions);

        currentOptions.setKeepLogFileNum(10);
        currentOptions.setInfoLogLevel(InfoLogLevel.WARN_LEVEL);
        currentOptions.setStatsDumpPeriodSec(0);
        currentOptions.setMaxLogFileSize(1024 * 1024); // 1 MB each

        return currentOptions;
    }

However, the rotating logger does currently not work (it will not delete old log files - see https://github.com/dataArtisans/frocksdb/pull/12). Also, the user should not have to write his own OptionsFactory to get a sensible default.

To prevent this file from filling up the disk, I propose to change Flink's default RocksDB settings so that the LOG file is effectively disabled (nothing is written to it by default).

Issue Links

• is related to: FLINK-23791 Enable RocksDB log again
• is related to: FLINK-14482 Bump up rocksdb version
• is related to: FLINK-23812 Support configuration of the RocksDB info logging via configuration
• is related to: FLINK-15747 Enable setting RocksDB log level from configuration
• links to: GitHub Pull Request #10437