[IGNITE-7382] Unknown Pair exception if work directory is removed after cluster shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3
• Fix Version/s: 2.8

Environment

RHEL 7.4
Docker

• Assignee: Unassigned
• Reporter: Sunny Chan
• Created: 11/Jan/18 07:42
• Updated: 10/Oct/19 13:11
• Resolved: 29/Apr/19 02:05

Description

To reproduce, try this:

1) Start a server node
2) Start a node in client mode, connect to server node
3) shutdown the server node and leave the client node running
4) remove Ignite work directory for the shutdown server node
5) Client node reconnects automatically, send a service call request
6) Exception occurs on Server, unable to deserialize org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2

2018-01-05 00:00:26.027 processors.job.GridJobWorker [svc-#273%clog%] ERROR - Failed to initialize job [jobId=839fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7, ses=GridJobSessionImpl [ses=GridTaskSessionImpl [taskName=o.a.i.i.processors.
service.GridServiceProxy$ServiceProxyCallable, dep=LocalDeployment [super=GridDeployment [ts=1515076515280, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, clsLdrId=3e4f891c061-5235b03b-87c3-4c29-a576-1b44936b0c11, user
Ver=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, undeployed=false, usage=0]], taskClsName=o.a.i.i.processors.service.GridServiceProxy$ServiceProxyCallable, sesId=739fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7, st
artTime=1515078026015, endTime=1515078036021, taskNodeId=ad77f485-d677-4694-8d0c-e780f16be9b7, clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false
, subjId=ad77f485-d677-4694-8d0c-e780f16be9b7, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=360535376]], execName=null], jobId=839fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7]]
org.apache.ignite.IgniteCheckedException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9859) [logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:438) [logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1109) [logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1913) [logic.jar:1.1.0]
        at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) [logic.jar:1.1.0]
        at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) [logic.jar:1.1.0]
        at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) [logic.jar:1.1.0]
        at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) [logic.jar:1.1.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to deserialize object [typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:874) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9853) [logic.jar:1.1.0]
        ... 10 more
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal object with optimized marshaller
        at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1786) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:833) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9853) ~[logic.jar:1.1.0]
        ... 10 more
Caused by: org.apache.ignite.IgniteCheckedException: Failed to find class with given class loader for unmarshalling (make sure same versions of all classes are available on all nodes or enable peer-class-loading) [clsLdr=sun.misc.Launche
r$AppClassLoader@764c12b6, cls=Unknown pair [platformId=0, typeId=873412491]]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:232) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1783) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:833) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9853) ~[logic.jar:1.1.0]
        ... 10 more
Caused by: java.lang.ClassNotFoundException: Unknown pair [platformId=0, typeId=873412491]
        at org.apache.ignite.internal.MarshallerContextImpl.getClassName(MarshallerContextImpl.java:376) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.MarshallerContextImpl.getClass(MarshallerContextImpl.java:326) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:268) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readClass(OptimizedObjectInputStream.java:372) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:323) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[logic.jar:1.1.0]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416) ~[?:1.8.0_121]
        at org.apache.ignite.internal.util.IgniteUtils.readClassArray(IgniteUtils.java:1702) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.service.GridServiceProxy$ServiceProxyCallable.readExternal(GridServiceProxy.java:438) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:545) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:199) ~[logic.jar:1.1.0]
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:416) ~[?:1.8.0_121]
        at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:227) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1783) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryUtils.doReadObject(BinaryUtils.java:1799) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.readObject(BinaryReaderExImpl.java:1329) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2.readBinary(GridClosureProcessor.java:1872) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:833) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) ~[logic.jar:1.1.0]
        at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) ~[logic.jar:1.1.0]
        at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9853) ~[logic.jar:1.1.0]
        ... 10 more

Issue Links

• is duplicated by: IGNITE-7794 Marshaller mappings are not saved to disk on joining nodes