[FLINK-16013] List and map config options could not be parsed correctly

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.10.1, 1.11.0
• Component/s: Runtime / Configuration
• Labels: pull-request-available
• Assignee: Yang Wang
• Reporter: Yang Wang
• Created: 12/Feb/20 06:30
• Updated: 25/Feb/20 07:59
• Resolved: 25/Feb/20 07:59

Description

Currently, if a config option is List type and written to a flink-conf.yaml, it could not be parsed correctly when reloaded from yaml resource. The root cause is we use List#toString to save into the yaml resource. However, when we want to parse a List from a string, we use semicolon to split the value.

 

Also the Map, Duration type have the same problem.

 

The following is a unit test to reproduce this problem.

public void testWriteConfigurationAndReload() throws IOException {
 final File flinkConfDir = temporaryFolder.newFolder().getAbsoluteFile();
 final Configuration flinkConfig = new Configuration();
 final ConfigOption<List<String>> listConfigOption = ConfigOptions
  .key("test-list-string-key")
  .stringType()
  .asList()
  .noDefaultValue();
 final List<String> values = Arrays.asList("value1", "value2", "value3");
 flinkConfig.set(listConfigOption, values);
 assertThat(values, Matchers.containsInAnyOrder(flinkConfig.get(listConfigOption).toArray()));

 BootstrapTools.writeConfiguration(flinkConfig, new File(flinkConfDir, "flink-conf.yaml"));
 final Configuration loadedFlinkConfig = GlobalConfiguration.loadConfiguration(flinkConfDir.getAbsolutePath());
 assertThat(values, Matchers.containsInAnyOrder(loadedFlinkConfig.get(listConfigOption).toArray()));
}

Issue Links

• links to: GitHub Pull Request #11070