Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-18242

Custom OptionsFactory settings seem to have no effect on RocksDB

    XMLWordPrintableJSON

    Details

    • Release Note:
      Hide
      After FLINK-18242, the deprecated `OptionsFactory` and `ConfigurableOptionsFactory` classes are removed (not applicable for release-1.10), please use `RocksDBOptionsFactory` and `ConfigurableRocksDBOptionsFactory` instead. Please also recompile your application codes if any class extending `DefaultConfigurableOptionsFactory`.
      Show
      After FLINK-18242 , the deprecated `OptionsFactory` and `ConfigurableOptionsFactory` classes are removed (not applicable for release-1.10), please use `RocksDBOptionsFactory` and `ConfigurableRocksDBOptionsFactory` instead. Please also recompile your application codes if any class extending `DefaultConfigurableOptionsFactory`.

      Description

       When I configure a custom OptionsFactory for RocksDB like this (similarly by specifying it via the state.backend.rocksdb.options-factory configuration):

      Configuration globalConfig = GlobalConfiguration.loadConfiguration();
      String checkpointDataUri = globalConfig.getString(CheckpointingOptions.CHECKPOINTS_DIRECTORY);
      RocksDBStateBackend stateBackend = new RocksDBStateBackend(checkpointDataUri);
      stateBackend.setOptions(new DefaultConfigurableOptionsFactoryWithLog());
      env.setStateBackend((StateBackend) stateBackend);

      it seems to be loaded

      2020-06-10 12:54:20,720 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using predefined options: DEFAULT.
      2020-06-10 12:54:20,721 INFO  org.apache.flink.contrib.streaming.state.RocksDBStateBackend  - Using application-defined options factory: DefaultConfigurableOptionsFactoryWithLog{DefaultConfigurableOptionsFactory{configuredOptions={}}}. 

      but it seems like none of the options defined in there is actually used. Just as an example, my factory does set the info log level to INFO_LEVEL but this is what you will see in the created RocksDB instance:

      > cat /tmp/flink-io-c95e8f48-0daa-4fb9-a9a7-0e4fb42e9135/*/db/OPTIONS*|grep info_log_level
        info_log_level=HEADER_LEVEL
        info_log_level=HEADER_LEVEL

      Together with the bug from FLINK-18241, it seems I cannot re-activate the RocksDB log that we disabled in FLINK-15068. FLINK-15747 was aiming at changing that particular configuration, but the problem seems broader since setDbLogDir() was actually also ignored and Flink itself does not change that setting.

        Attachments

          Activity

            People

            • Assignee:
              liyu Yu Li
              Reporter:
              NicoK Nico Kruber
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: