[IGNITE-17481] Ignite shutdown sequence throws a ClassCastException from inside GridManagerAdapter on latest Java 11.0.16 and 17.0.4 point releases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 2.10, 2.12, 2.13
• Fix Version/s: 2.14
• Component/s: general
• Release Note: Fixed ClassCastException that appears during deployment manager stop in Java versions 11.0.16+ and 17.0.4+
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Ivan Bessonov
• Reporter: Paolo de Dios
• Reviewer: Aleksandr Polovtsev
• Created: 07/Aug/22 05:28
• Updated: 29/Aug/22 09:35
• Resolved: 26/Aug/22 07:22

Description

 

When ClassLoaders are undeployed, the `GridDeploymentStoreAdapter.clearSerializationCache()` method attempts to clear serialization caches to avoid PermGen memory leaks.  The implementation of this optimization seems to no longer work as the underlying JVM implementaiton of `java.io.ObjectInputStream$Caches` and java.io.ObjectOutputStream$Caches` no longer maintain a private cache of subclass security audit results as a java.util.Map, which Ignite expects inside `GridDeploymentStoreAdapter.clearSerializationCache()`.

 

Stacktrace

 

[INFO ] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[GridDeploymentLocalStore] - Removed undeployed class: GridDeployment [ts=1659817673460, depMode=SHARED, clsLdr=jdk.internal.loader.ClassLoaders$AppClassLoader@277050dc, clsLdrId=b2497d47281-7ff6d972-ec5d-4d9c-bc60-95463b5e10b6, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, pendingUndeploy=false, undeployed=true, usage=0]
[ERROR] 2022-08-06T20:28:04,778+0000 T=[vert.x-eventloop-thread-4] L=[local] - Failed to stop component (ignoring): GridManagerAdapter [enabled=true, name=o.a.i.i.managers.deployment.GridDeploymentManager]
java.lang.ClassCastException: class java.io.ObjectInputStream$Caches$1 cannot be cast to class java.util.Map (java.io.ObjectInputStream$Caches$1 and java.util.Map are in module java.base of loader 'bootstrap')
        at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCache(GridDeploymentStoreAdapter.java:151) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.managers.deployment.GridDeploymentStoreAdapter.clearSerializationCaches(GridDeploymentStoreAdapter.java:120) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.undeploy(GridDeploymentLocalStore.java:565) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.managers.deployment.GridDeploymentLocalStore.stop(GridDeploymentLocalStore.java:101) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.storesStop(GridDeploymentManager.java:630) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.managers.deployment.GridDeploymentManager.stop(GridDeploymentManager.java:137) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1928) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1806) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2382) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2205) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:350) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at org.apache.ignite.Ignition.stop(Ignition.java:230) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.appliedtheory.disco.services.IgniteClusterBootstrap.stop(IgniteClusterBootstrap.java:1148) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.appliedtheory.disco.services.IgniteClusterService.doStop(IgniteClusterService.java:255) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.appliedtheory.core.component.AbstractComponent.stop(AbstractComponent.java:43) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.appliedtheory.framework.container.ServiceContainerEngine.lambda$doClose$1(ServiceContainerEngine.java:220) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.reactivex.rxjava3.internal.observers.AbstractDisposableAutoRelease.onComplete(AbstractDisposableAutoRelease.java:89) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.vertx.rxjava3.impl.AsyncResultCompletable.lambda$subscribeActual$0(AsyncResultCompletable.java:55) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.vertx.core.impl.VertxImpl$1$1.lambda$operationComplete$0(VertxImpl.java:860) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[proof-web-gateway-1.0.0+4dbe618b49758c4e-aio.jar:1.0.0]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
 

As detailed by the commit references below, recent versions of Java 11 and 17 have changed the underlying implementation of `java.io.ObjectInputStream$Caches` from using a  `java.util.concurrent.ConcurrentMap` to a `java.lang.ClassValue`.

JDK 17 change

https://github.com/openjdk/jdk17/commit/8fed8ab29cae4f189f44609c23f116967eef6bdf

JDK 11 change

https://github.com/openjdk/jdk11u/commit/01c4d3fec62d198b2575e1b4480eb8e659776c56

 

Current Workaround: 

Pin Java 17 to 17.0.3 and earlier.

Pin Java 11 to 11.0.15 and earlier

 

Issue Links

• duplicates: IGNITE-17549 Error on Ignite.close(): class java.io.ObjectInputStream$Caches$1 cannot be cast to class java.util.Map
• links to: GitHub Pull Request #10216
• links to: User list: Error on Ignite.close(): class java.io.ObjectInputStream$Caches$1 cannot be cast to class java.util.Map