[FLINK-36810] Reflective access warning not addressed causing Flink failing to process job creation

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.20.0
• Component/s: API / DataStream
• Assignee: Unassigned
• Reporter: Yusu Gao
• Created: 27/Nov/24 15:07
• Updated: 27/Nov/24 16:24

Description

While we are testing migration our Flink cluster from 1.11 on Java 11 runtime to 1.20 on Java 17 runtime, we noticed a job creation error.

"errors":["org.apache.flink.runtime.rest.handler.RestHandlerException: Could not execute application.
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$1(JarRunHandler.java:114)
    at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
    at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.util.concurrent.CompletionException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.datastream.impl.ExecutionContextEnvironment
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770)
    ... 1 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.datastream.impl.ExecutionContextEnvironment
    at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:109)
    at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java:84)
    at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java:70)
    at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$0(JarRunHandler.java:108)
    at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
    ... 1 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread \"flink-jar-runner-thread-1\"]
    ... 6 more 

Reverting runtime back to Java 11 with --illegal-access=warn, we found a warning for Flink code

WARNING: Illegal reflective access by org.apache.flink.streaming.runtime.translators.DataStreamV2SinkTransformationTranslator (file:/path/to/flink/flink-1.20.0/lib/flink-dist-1.20.0.jar) to field java.util.Collections$UnmodifiableMap.m 

Which points to DataStreamV2SinkTransformationTranslator which is used exactly in the class throwing exception.

And adding argument to the JVM_ARGS env var fixed issue

export JVM_ARGS="$JVM_ARGS --add-opens=java.base/java.util=ALL-UNNAMED"

This is a workaround but not necessarily the best solution nor the safest. Please help provide insights or better solution for such issue.