[IGNITE-23202] Services: node crashes when node filter class was not found in class path

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.16
• Fix Version/s: 2.17
• Component/s: managed services
• Labels: ise
• Release Note: Fixed server node fail when node service filter class unknown
• Ignite Flags: Release Notes Required
• Assignee: Nikita Amelchev
• Reporter: Ilya Shishkov
• Created: 12/Sep/24 17:00
• Updated: 07/Oct/24 15:13
• Resolved: 07/Oct/24 15:12

Description

How to reproduce:

1. Start node(s) (eg. via ignite.sh)
2. Start another node in order to deploy service with service node filter which is not present in classpath of server nodes. As example see patch of ServicesExample.java: IGNITE-23202.patch
3. Some nodes (may be all) will fail.

Stacktraces differ for anonymous class and lamda:

Lambda

[20:36:56,908][INFO][exchange-worker-#53][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=true, evt=NODE_FAILED, evtNode=97ae84a3-094e-47b7-92db-8f059f8a2512, customEvt=null, allowMerge=true, exchangeFreeSwitch=false]
[20:36:56,909][SEVERE][tcp-disco-msg-worker-[97ae84a3 0:0:0:0:0:0:0:1%lo:47500]-#2-#45][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=a952f13f-349e-4a69-bff8-cbcaeca9b1eb, id=2213f47e191-34c6795a-e1fe-40d6-b45d-229bbe2c358d, verifierNodeId=a952f13f-349e-4a69-bff8-cbcaeca9b1eb, topVer=4, pendingIdx=0, failedNodes=null, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=a952f13f-349e-4a69-bff8-cbcaeca9b1eb, id=2213f47e191-34c6795a-e1fe-40d6-b45d-229bbe2c358d, verifierNodeId=a952f13f-349e-4a69-bff8-cbcaeca9b1eb, topVer=4, pendingIdx=0, failedNodes=null, isClient=false]]
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6307)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:6097)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3237)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2892)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7980)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3066)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7911)
	at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to deserialize object with given class loader: jdk.internal.loader.ClassLoaders$AppClassLoader@55054057
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:132)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:139)
	at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
	at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10807)
	at org.apache.ignite.spi.discovery.tcp.messages.TcpDiscoveryCustomEventMessage.message(TcpDiscoveryCustomEventMessage.java:103)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6304)
	... 8 more
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.ignite.services.ServiceConfiguration.nodeFilter of type org.apache.ignite.lang.IgnitePredicate in instance of org.apache.ignite.internal.processors.service.LazyServiceConfiguration
	at java.base/java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2076)
	at java.base/java.io.ObjectStreamClass$FieldReflector.checkObjectFieldValueTypes(ObjectStreamClass.java:2039)
	at java.base/java.io.ObjectStreamClass.checkObjFieldValueTypes(ObjectStreamClass.java:1293)
	at java.base/java.io.ObjectInputStream.defaultCheckFieldValues(ObjectInputStream.java:2512)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2419)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
	at java.base/java.util.ArrayList.readObject(ArrayList.java:929)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2357)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:124)
	... 13 more

Anonymous class

[20:40:54,949][SEVERE][tcp-disco-msg-worker-[87a727e0 0:0:0:0:0:0:0:1%lo:47501 crd]-#2-#45][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=1c2bff3c-72c4-4cfc-832c-50bda9af1b10, id=4b4d257e191-1c2bff3c-72c4-4cfc-832c-50bda9af1b10, verifierNodeId=7146786b-cd3d-4ab7-aa7f-28293a1e1de8, topVer=3, pendingIdx=0, failedNodes=null, isClient=false]]]]
class org.apache.ignite.IgniteException: Failed to unmarshal discovery custom message: TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=1c2bff3c-72c4-4cfc-832c-50bda9af1b10, id=4b4d257e191-1c2bff3c-72c4-4cfc-832c-50bda9af1b10, verifierNodeId=7146786b-cd3d-4ab7-aa7f-28293a1e1de8, topVer=3, pendingIdx=0, failedNodes=null, isClient=false]]
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6307)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:6097)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3237)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2892)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7980)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3066)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7911)
	at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@55054057, cls=org.apache.ignite.examples.servicegrid.ServicesExample$1]
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:129)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:139)
	at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
	at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:10807)
	at org.apache.ignite.spi.discovery.tcp.messages.TcpDiscoveryCustomEventMessage.message(TcpDiscoveryCustomEventMessage.java:103)
	at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6304)
	... 8 more
Caused by: java.lang.ClassNotFoundException: org.apache.ignite.examples.servicegrid.ServicesExample$1
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9349)
	at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9287)
	at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.resolveClass(JdkMarshallerObjectInputStream.java:59)
	at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2003)
	at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1870)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2201)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
	at java.base/java.util.ArrayList.readObject(ArrayList.java:929)
	at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.base/java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1046)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2357)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2496)
	at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2390)
	at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2228)
	at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1687)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:489)
	at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:447)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:124)
	... 13 more

Issue Links

• is related to: IGNITE-23372 Revise validation of received cluster data by a joining node.
• relates to: IGNITE-23369 Unable to deploy a service if node filter class is present only on cluster group.
• relates to: IGNITE-23368 Refactor processing of LazyServiceConfiguration
• links to: GitHub Pull Request #11539