Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-9090

When client node make cache.QueryCursorImpl.getAll they have OOM and continue working

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.4
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
      None
    • Environment:

      2 server node, 1 client, 1 cache with 15 kk size

      Description

      [12:21:22,390][SEVERE][query-#69][GridCacheIoManager] Failed to process message [senderId=30cab4ec-1da7-4e9f-a262-bdfa4d466865, messageType=class o.a.i.i.processors.cache.query.GridCacheQueryResponse]
      java.lang.OutOfMemoryError: GC overhead limit exceeded
              at java.lang.Long.valueOf(Long.java:840)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:250)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:421)
              at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(OptimizedObjectInputStream.java:555)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(OptimizedClassDescriptor.java:917)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0(OptimizedObjectInputStream.java:346)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:198)
              at java.io.ObjectInputStream.readObject(ObjectInputStream.java:421)
              at org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(OptimizedMarshaller.java:227)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
              at org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1777)
              at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1964)
              at org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1716)
              at org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310)
              at org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
              at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
              at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.unmarshalCollection0(GridCacheQueryResponse.java:189)
              at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponse.finishUnmarshal(GridCacheQueryResponse.java:162)
              at org.apache.ignite.internal.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:1530)
              at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:576)
              at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$700(GridCacheIoManager.java:101)
              at org.apache.ignite.internal.processors.cache.GridCacheIoManager$OrderedMessageListener.onMessage(GridCacheIoManager.java:1613)
              at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
              at org.apache.ignite.internal.managers.communication.GridIoManager.access$4100(GridIoManager.java:125)
              at org.apache.ignite.internal.managers.communication.GridIoManager$GridCommunicationMessageSet.unwind(GridIoManager.java:2752)
              at org.apache.ignite.internal.managers.communication.GridIoManager.unwindMessageSet(GridIoManager.java:1516)
              at org.apache.ignite.internal.managers.communication.GridIoManager.access$4400(GridIoManager.java:125)
              at org.apache.ignite.internal.managers.communication.GridIoManager$10.run(GridIoManager.java:1485)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      [12:21:28,573][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available.
      [12:21:23,759][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 22446 milliseconds.
      [12:21:23,758][INFO][grid-timeout-worker-#39][IgniteKernal]
      Metrics for local node (to disable set 'metricsLogFrequency' to 0)
          ^-- Node [id=c1f087b1, uptime=00:01:25.431]
          ^-- H/N/C [hosts=2, nodes=3, CPUs=32]
          ^-- CPU [cur=100%, avg=79.09%, GC=8.93%]
          ^-- PageMemory [pages=0]
          ^-- Heap [used=216MB, free=8.57%, comm=236MB]
          ^-- Non heap [used=62MB, free=-1%, comm=65MB]
          ^-- Outbound messages queue [size=0]
          ^-- Public thread pool [active=0, idle=0, qSize=0]
          ^-- System thread pool [active=0, idle=0, qSize=0]
      [12:21:32,609][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 8800 milliseconds.
      [12:21:36,042][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3383 milliseconds.
      [12:21:41,883][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 5791 milliseconds.
      [12:21:41,884][SEVERE][grid-timeout-worker-#39][GridTimeoutProcessor] Error when executing timeout callback: CancelableTask [id=92823e5d461-bae41fe7-1702-4a25-8681-916897029985, endTime=1532596959200, period=60000, cancel=false, task=o.a.i.i.IgniteKernal$4@c0ec5ba]
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      [12:21:43,225][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1291 milliseconds.
      [12:21:44,553][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1279 milliseconds.
      [12:21:46,979][SEVERE][grid-timeout-worker-#39][GridTimeoutProcessor] Runtime error caught during grid runnable execution: GridWorker [name=grid-timeout-worker, igniteInstanceName=null, finished=false, hashCode=1065113566, interrupted=false, runner=grid-timeout-worker-#39]
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      [12:21:48,392][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3789 milliseconds.
      [12:21:49,712][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1270 milliseconds.
      [12:21:51,007][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1245 milliseconds.
      [12:21:52,115][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1058 milliseconds.
      [12:21:53,427][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1262 milliseconds.
      [12:21:53,427][SEVERE][tcp-client-disco-sock-reader-#3][TcpDiscoverySpi] Runtime error caught during grid runnable execution: IgniteSpiThread [name=tcp-client-disco-sock-reader-#3]
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      [12:21:54,808][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1331 milliseconds.
      [12:21:55,594][SEVERE][ignite-license-checker-null][GridEntLicenseProcessor] License grace/burst period - left 57 mins.
      [12:22:18,203][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 1368 milliseconds.
      [12:22:25,228][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 2370 milliseconds.
      [12:22:28,428][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 3150 milliseconds.
      [12:22:32,748][WARNING][jvm-pause-detector-worker][] Possible too long JVM pause: 4270 milliseconds.
      [12:22:37,250][SEVERE][tcp-client-disco-sock-writer-#2][TcpDiscoverySpi] Failed to send message: null
      java.io.IOException: Failed to get acknowledge for message: TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=cb823e5d461-c1f087b1-678e-4377-a3d1-4bb5dbdb4c98, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=true]]
              at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1359)
              at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              ARomantsov ARomantsov
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: