[FLINK-29835] NoClassDefFoundError in PulsarSourceUnorderedE2ECase

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Invalid
• Affects Version/s: 1.16.0, 1.15.3, pulsar-3.0.0, pulsar-4.0.0
• Component/s: Connectors / Pulsar, Runtime / Coordination
• Labels: test-stability
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 01/Nov/22 09:36
• Updated: 04/Jan/23 02:53
• Resolved: 04/Jan/23 02:53

Description

This build failed in PulsarSourceUnorderedE2ECase.testSavepoint due to some job timeout.

The logs reveal NoClassDefFoundErrors:

2022-11-14 15:36:59,696 WARN  org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient [] - Unexpected error on ChannelManager close
java.lang.NoClassDefFoundError: org/apache/pulsar/shade/io/netty/util/concurrent/DefaultPromise$1
     at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.ensureThreadStarted(SingleThreadEventExecutor.java:970) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.shutdownGracefully(SingleThreadEventExecutor.java:661) ~[?:?]
     at org.apache.pulsar.shade.io.netty.util.concurrent.MultithreadEventExecutorGroup.shutdownGracefully(MultithreadEventExecutorGroup.java:163) ~[?:?]
     at org.apache.pulsar.shade.org.asynchttpclient.netty.channel.ChannelManager.close(ChannelManager.java:307) ~[?:?]
     at org.apache.pulsar.shade.org.asynchttpclient.DefaultAsyncHttpClient.close(DefaultAsyncHttpClient.java:120) ~[?:?]
     at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.close(AsyncHttpConnector.java:346) ~[?:?]
     at org.apache.pulsar.client.admin.internal.PulsarAdminImpl.close(PulsarAdminImpl.java:490) ~[?:?]
     at org.apache.flink.connector.pulsar.source.enumerator.PulsarSourceEnumerator.close(PulsarSourceEnumerator.java:172) ~[?:?]
     at org.apache.flink.util.IOUtils.closeAll(IOUtils.java:255) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at org.apache.flink.runtime.source.coordinator.SourceCoordinator.close(SourceCoordinator.java:265) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at org.apache.flink.runtime.operators.coordination.ComponentClosingUtils.lambda$closeAsyncWithTimeout$0(ComponentClosingUtils.java:76) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: java.lang.ClassNotFoundException: org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise$1
     at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_342]
     at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
     at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) ~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
     at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
     ... 16 more 

Outdated:

The issue seems to be related to the BlobServer failing to provide some artifacts (java.io.FileNotFoundException) that consequently causes classes not being found.

 

Issue Links

• causes: FLINK-30351 Enable test cases again
• Discovered while testing: FLINK-29755 PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers