[IGNITE-13642] Nodes fail when they meet objects of unknown type in metastorage

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 2.9
• Labels: 2.9.1-rc
• Ignite Flags: Release Notes Required
• Assignee: Unassigned
• Reporter: Denis Mekhanikov
• Created: 29/Oct/20 10:40
• Updated: 31/Mar/21 06:53
• Resolved: 31/Mar/21 06:53

Description

When a node sees an object of a class that is missing on this node's classpath, it fails with the following exception:

[16:46:47,134][SEVERE][disco-notifier-worker-#41][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, cls=example.ClientNode$BamboozleClass]]]
class org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, cls=example.ClientNode$BamboozleClass]
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:128)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:138)
	at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:80)
	at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageUtil.unmarshal(DistributedMetaStorageUtil.java:61)
	at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.completeWrite(DistributedMetaStorageImpl.java:1161)
	at org.apache.ignite.internal.processors.metastorage.persistence.DistributedMetaStorageImpl.onUpdateMessage(DistributedMetaStorageImpl.java:1089)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:650)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:521)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2718)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2756)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: example.ClientNode$BamboozleClass
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.ignite.internal.util.IgniteUtils.forName(IgniteUtils.java:9061)
	at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.resolveClass(JdkMarshallerObjectInputStream.java:58)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1925)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1808)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2099)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
	at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:123)
	... 11 more

The result is that one node can write an object of some custom class to the metastorage and make all other nodes fail.

The following reproducer can be used:

public class ClientNode {
    public static void main(String[] args) throws IgniteCheckedException {
        IgniteConfiguration igniteCfg = nodeConfiguration().setClientMode(true);

        IgniteKernal ignite = (IgniteKernal) Ignition.start(igniteCfg);
        DistributedMetaStorage metaStorage = ignite.context().distributedMetastorage();

        metaStorage.write("hey", new BamboozleClass());
    }

    private static IgniteConfiguration nodeConfiguration() {
        IgniteConfiguration igniteCfg = new IgniteConfiguration();

        TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
        discoverySpi.setIpFinder(new TcpDiscoveryVmIpFinder()
                .setAddresses(Collections.singleton("127.0.0.1:47500..47600")));

        igniteCfg.setDiscoverySpi(discoverySpi);

        return igniteCfg;
    }

    public static class BamboozleClass implements Serializable {
    }
}

Issue Links

• duplicates: IGNITE-14347 Fix Node failure on Receiving Data of Unknown class via Distributed Metastorage