Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20787 Improve the Table API to make it usable
  3. FLINK-18019

The configuration specified in TableConfig may not take effect in certain cases

    XMLWordPrintableJSON

    Details

      Description

      Currently If the following configuration is configured in flink-conf.yaml:

      state.backend: rocksdb
      state.checkpoints.dir: file:///tmp/flink-checkpoints
      

      and the following configuration is configured via TableConfig:

      tableConfig.getConfiguration().setString("state.backend.rocksdb.memory.fixed-per-slot", "200MB")
      tableConfig.getConfiguration().setString("taskmanager.memory.task.off-heap.size", "200MB")
      

      Then users submit the job via CliFrontend, the configuration set via TableConfig will not take effect.

      Intuitively, it should be that user specified configuration via TableConfig(has higher priority) and the configuration specified via flink-conf.yaml together determines the configuration of a job. However, it doesn't hold in all cases.

      The root cause is that only the configuration specified in TableConfig in passed to StreamExecutionEnvironment during translate to plan. For the above case, as state.backend is not specified in TableConfig and so the configuration state.backend.rocksdb.memory.fixed-per-slot will not take effect. Please note that in above example, the state backend actually used will be RocksDB without the configurationĀ state.backend.rocksdb.memory.fixed-per-slot andĀ taskmanager.memory.task.off-heap.size.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dian.fu Dian Fu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: