[FLINK-18242] Custom OptionsFactory settings seem to have no effect on RocksDB - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.10.0, 1.10.1, 1.11.0
Fix Version/s: 1.10.2, 1.11.0, 1.12.0
Component/s: Runtime / State Backends
Labels:
- pull-request-available

Release Note:

Hide
After ~~FLINK-18242~~, the deprecated `OptionsFactory` and `ConfigurableOptionsFactory` classes are removed (not applicable for release-1.10), please use `RocksDBOptionsFactory` and `ConfigurableRocksDBOptionsFactory` instead. Please also recompile your application codes if any class extending `DefaultConfigurableOptionsFactory`.

Show
After FLINK-18242 , the deprecated `OptionsFactory` and `ConfigurableOptionsFactory` classes are removed (not applicable for release-1.10), please use `RocksDBOptionsFactory` and `ConfigurableRocksDBOptionsFactory` instead. Please also recompile your application codes if any class extending `DefaultConfigurableOptionsFactory`.

Description

When I configure a custom OptionsFactory for RocksDB like this (similarly by specifying it via the state.backend.rocksdb.options-factory configuration):

Configuration globalConfig = GlobalConfiguration.loadConfiguration();
String checkpointDataUri = globalConfig.getString(CheckpointingOptions.CHECKPOINTS_DIRECTORY);
RocksDBStateBackend stateBackend = new RocksDBStateBackend(checkpointDataUri);
stateBackend.setOptions(new DefaultConfigurableOptionsFactoryWithLog());
env.setStateBackend((StateBackend) stateBackend);

it seems to be loaded

2020-06-10 12:54:20,720 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using predefined options: DEFAULT.
2020-06-10 12:54:20,721 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using application-defined options factory: DefaultConfigurableOptionsFactoryWithLog{DefaultConfigurableOptionsFactory{configuredOptions={}}}.

but it seems like none of the options defined in there is actually used. Just as an example, my factory does set the info log level to INFO_LEVEL but this is what you will see in the created RocksDB instance:

> cat /tmp/flink-io-c95e8f48-0daa-4fb9-a9a7-0e4fb42e9135/*/db/OPTIONS*|grep info_log_level
  info_log_level=HEADER_LEVEL
  info_log_level=HEADER_LEVEL

Together with the bug from ~~FLINK-18241~~, it seems I cannot re-activate the RocksDB log that we disabled in ~~FLINK-15068~~. ~~FLINK-15747~~ was aiming at changing that particular configuration, but the problem seems broader since setDbLogDir() was actually also ignored and Flink itself does not change that setting.