[FLINK-36454] FlinkContainer always overwrites default config

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0-preview
• Component/s: Test Infrastructure
• Assignee: Unassigned
• Reporter: Arvid Heise
• Created: 09/Oct/24 12:10
• Updated: 09/Oct/24 12:10

Description

Because FlinkContainer disregards the default config, some meaningful options are lost when executing container in tests. For example, because we overwrite env.java.opts.all, all --add-opens are lost leading to exceptions like

 10:26:26,171 [                main] ERROR org.apache.flink.connector.testframe.container.FlinkContainers [] - java.lang.ExceptionInInitializerError
	at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:109)
	at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:1026)
	at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:247)
	at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1270)
	at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$10(CliFrontend.java:1367)
	at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
	at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1367)
	at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1335)
Caused by: java.lang.RuntimeException: Can not register process function transformation translator.
	at org.apache.flink.datastream.impl.ExecutionEnvironmentImpl.<clinit>(ExecutionEnvironmentImpl.java:98)
	... 8 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private final java.util.Map java.util.Collections$UnmodifiableMap.m accessible: module java.base does not "opens java.util" to unnamed module @771b8d5c
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)
	at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)
	at org.apache.flink.streaming.runtime.translators.DataStreamV2SinkTransformationTranslator.registerSinkTransformationTranslator(DataStreamV2SinkTransformationTranslator.java:104)
	at org.apache.flink.datastream.impl.ExecutionEnvironmentImpl.<clinit>(ExecutionEnvironmentImpl.java:96)
	... 8 more 

FlinkImageBuilder should read the default config and only overwrite the custom settings in

private Path createTemporaryFlinkConfFile(Configuration finalConfiguration, Path tempDirectory)
        throws IOException {
    Path flinkConfFile = tempDirectory.resolve(GlobalConfiguration.FLINK_CONF_FILENAME);
    Files.write(
            flinkConfFile,
            ConfigurationUtils.convertConfigToWritableLines(finalConfiguration, false));

    return flinkConfFile;
} 

The workaround is to set the option manually in the test but that may be outdated on version upgrade.