Details
-
Bug
-
Status: Open
-
Critical
-
Resolution: Unresolved
-
2.4
-
None
-
None
-
2 server node, 1 client, 1 cache with 15 kk size
Description
[12:21:22,390][SEVERE][query-#69][GridCacheIoManager] Failed to process message [senderId=30cab4ec-1da7-4e9f-a262-bdfa4d466865, messageType=class o.a.i.i.processors.cache.query.GridCacheQueryResponse] java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Long.valueOf(Long.java:840) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:250) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:421) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555) at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346) at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:421) at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:227) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1777) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964) at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716) at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310) at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.unmarshalCollection0(GridCacheQueryResponse.java:189) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.finishUnmarshal(GridCacheQueryResponse.java:162) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1530) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:576) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1613) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2752) at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1516) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:125) at org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1485) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [12:21:28,573][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [12:21:23,759][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 22446 milliseconds. [12:21:23,758][INFO][grid-timeout-worker-#39][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=c1f087b1, uptime=00:01:25.431] ^-- H/N/C [hosts=2, nodes=3, CPUs=32] ^-- CPU [cur=100%, avg=79.09%, GC=8.93%] ^-- PageMemory [pages=0] ^-- Heap [used=216MB, free=8.57%, comm=236MB] ^-- Non heap [used=62MB, free=-1%, comm=65MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=0, qSize=0] [12:21:32,609][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 8800 milliseconds. [12:21:36,042][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3383 milliseconds. [12:21:41,883][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 5791 milliseconds. [12:21:41,884][SEVERE][grid-timeout-worker-#39][GridTimeoutProcessor] Error when executing timeout callback: CancelableTask [id=92823e5d461-bae41fe7-1702-4a25-8681-916897029985, endTime=1532596959200, period=60000, cancel=false, task=o.a.i.i.IgniteKernal$4@c0ec5ba] java.lang.OutOfMemoryError: GC overhead limit exceeded [12:21:43,225][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1291 milliseconds. [12:21:44,553][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1279 milliseconds. [12:21:46,979][SEVERE][grid-timeout-worker-#39][GridTimeoutProcessor] Runtime error caught during grid runnable execution: GridWorker [name=grid-timeout-worker, igniteInstanceName=null, finished=false, hashCode=1065113566, interrupted=false, runner=grid-timeout-worker-#39] java.lang.OutOfMemoryError: GC overhead limit exceeded [12:21:48,392][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3789 milliseconds. [12:21:49,712][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1270 milliseconds. [12:21:51,007][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1245 milliseconds. [12:21:52,115][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1058 milliseconds. [12:21:53,427][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1262 milliseconds. [12:21:53,427][SEVERE][tcp-client-disco-sock-reader-#3][TcpDiscoverySpi] Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-client-disco-sock-reader-#3] java.lang.OutOfMemoryError: GC overhead limit exceeded [12:21:54,808][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1331 milliseconds. [12:21:55,594][SEVERE][ignite-license-checker-null][GridEntLicenseProcessor] License grace/burst period - left 57 mins. [12:22:18,203][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1368 milliseconds. [12:22:25,228][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 2370 milliseconds. [12:22:28,428][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3150 milliseconds. [12:22:32,748][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 4270 milliseconds. [12:22:37,250][SEVERE][tcp-client-disco-sock-writer-#2][TcpDiscoverySpi] Failed to send message: null java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=cb823e5d461-c1f087b1-678e-4377-a3d1-4bb5dbdb4c98, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]] at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1359) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)