[FLINK-30299] TaskManagerRunnerTest fails with 239 exit code (i.e. FatalExitExceptionHandler was called) NoClassDefFoundError: akka/remote/transport/netty/NettyFutureBridge$$anon$1

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.16.0, 1.17.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available test-stability
• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 05/Dec/22 10:28
• Updated: 13/Jun/23 21:32
• Resolved: 13/Jun/23 21:32

Description

We're again experiencing 239 exit code being caused by FatalExitExceptionHandler due class loading issues:

04:53:03,365 [flink-akka.remote.default-remote-dispatcher-8] ERROR org.apache.flink.util.FatalExitExceptionHandler              [] - FATAL: Thread 'flink-akka.remote.default-remote-dispatcher-8' produced an uncaught exception. Stopping the process...
java.lang.NoClassDefFoundError: akka/remote/transport/netty/NettyFutureBridge$$anon$1
        at akka.remote.transport.netty.NettyFutureBridge$.apply(NettyTransport.scala:65) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.remote.transport.netty.NettyTransport.$anonfun$associate$1(NettyTransport.scala:566) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:303) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:37) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49) ~[flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48) [flink-rpc-akka_b340b753-81f5-4e09-b083-5f8c92589fad.jar:1.16-SNAPSHOT]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) [?:1.8.0_292]
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) [?:1.8.0_292]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) [?:1.8.0_292]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) [?:1.8.0_292]
Caused by: java.lang.ClassNotFoundException: akka.remote.transport.netty.NettyFutureBridge$$anon$1
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_292]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_292]
        at org.apache.flink.core.classloading.ComponentClassLoader.loadClassFromComponentOnly(ComponentClassLoader.java:149) ~[flink-core-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
        at org.apache.flink.core.classloading.ComponentClassLoader.loadClass(ComponentClassLoader.java:112) ~[flink-core-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_292]
        ... 16 more

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43694&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=8319

I created this as a follow-up of FLINK-26037 becasue we repurposed it and fixed a bug in FLINK-26037. But it looks like both are being caused by the same issue.

Issue Links

• is duplicated by: FLINK-30140 flink-runtime intergration tests fail due to Process Exit Code: 239 (because of NoClassDefFoundError akka.remote.transport.netty.NettyFutureBridge$$anon$1)
• is related to: FLINK-26037 TaskManagerRunner may crash during shutdown sequence
• relates to: FLINK-31669 DynamicFilteringITCase failed fataly because of NoClassDefFoundError of akka/actor/SuppressedDeadLetter
• links to: GitHub Pull Request #21463
• links to: GitHub Pull Request #21533
• links to: GitHub Pull Request #21534