Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-1924

Incomplete marshaller cache rebalancing causes Grid hangs under SSL

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8
    • Component/s: None
    • Labels:

      Description

      End of the log.
      [11:49:32] : [org.apache.ignite:ignite-core] [11:49:32,947][INFO ]exchange-worker-#220584%tcp.IgniteCacheSslStartStopSelfTest3%[GridDhtPartitionDemander] <ignite-marshaller-sys-cache> Starting rebalancing [cache=ignite-marshaller-sys-cache, mode=SYNC, fromNode=108bffdb-1c1e-49aa-9525-b434784fa001, partitionsCount=7, topology=AffinityTopologyVersion [topVer=594, minorTopVer=0], updateSeq=1]
      [11:49:32] : [org.apache.ignite:ignite-core] [11:49:32,962][INFO ]exchange-worker-#220584%tcp.IgniteCacheSslStartStopSelfTest3%[GridDhtPartitionDemander] <ignite-marshaller-sys-cache> Starting rebalancing [cache=ignite-marshaller-sys-cache, mode=SYNC, fromNode=20660c29-91a1-4279-9dc1-88d192bc6002, partitionsCount=6, topology=AffinityTopologyVersion [topVer=594, minorTopVer=0], updateSeq=1]
      [11:49:32] : [org.apache.ignite:ignite-core] [11:49:32,962][INFO ]exchange-worker-#220584%tcp.IgniteCacheSslStartStopSelfTest3%[GridDhtPartitionDemander] <ignite-marshaller-sys-cache> Starting rebalancing [cache=ignite-marshaller-sys-cache, mode=SYNC, fromNode=00b3a75a-074d-46a5-a158-3956c0ec4000, partitionsCount=7, topology=AffinityTopologyVersion [topVer=594, minorTopVer=0], updateSeq=1]
      [11:49:32] : [org.apache.ignite:ignite-core] [11:49:32,963][INFO ]ignite-#220587%marshaller-cache-tcp.IgniteCacheSslStartStopSelfTest3%[GridDhtPartitionDemander] <ignite-marshaller-sys-cache> Completed rebalancing [cache=ignite-marshaller-sys-cache, fromNode=00b3a75a-074d-46a5-a158-3956c0ec4000, topology=AffinityTopologyVersion [topVer=594, minorTopVer=0], time=21 ms]
      [11:49:32] : [org.apache.ignite:ignite-core] [11:49:32,963][INFO ]ignite-#220586%marshaller-cache-tcp.IgniteCacheSslStartStopSelfTest3%[GridDhtPartitionDemander] <ignite-marshaller-sys-cache> Completed rebalancing [cache=ignite-marshaller-sys-cache, fromNode=108bffdb-1c1e-49aa-9525-b434784fa001, topology=AffinityTopologyVersion [topVer=594, minorTopVer=0], time=21 ms]

      Hang on:

      [11:51:56] : [org.apache.ignite:ignite-core] Thread name="ignite-#220562%sys-tcp.IgniteCacheSslStartStopSelfTest3%", id=287517, state=WAITING, blockCnt=0, waitCnt=3
      [11:51:56] : [org.apache.ignite:ignite-core] Lock [object=o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$RebalanceFuture@b402f89, ownerName=null, ownerId=-1]
      [11:51:56] : [org.apache.ignite:ignite-core] at sun.misc.Unsafe.park(Native Method)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:157)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.waitForCacheRebalancing(GridDhtPartitionDemander.java:265)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.access$400(GridDhtPartitionDemander.java:85)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$3.call(GridDhtPartitionDemander.java:323)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander$3.call(GridDhtPartitionDemander.java:320)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker$1.call(GridCachePartitionExchangeManager.java:1386)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker$1.call(GridCachePartitionExchangeManager.java:1377)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6371)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:929)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:745)
      [11:51:56] : [org.apache.ignite:ignite-core]
      [11:51:56] : [org.apache.ignite:ignite-core] Locked synchronizers:
      [11:51:56] : [org.apache.ignite:ignite-core] java.util.concurrent.ThreadPoolExecutor$Worker@7a9245cc
      [11:51:56] : [org.apache.ignite:ignite-core] Thread name="ignite-#220561%sys-tcp.IgniteCacheSslStartStopSelfTest3%", id=287516, state=WAITING, blockCnt=0, waitCnt=2
      [11:51:56] : [org.apache.ignite:ignite-core] Lock [object=java.util.concurrent.CountDownLatch$Sync@22f0d124, ownerName=null, ownerId=-1]
      [11:51:56] : [org.apache.ignite:ignite-core] at sun.misc.Unsafe.park(Native Method)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.IgniteUtils.awaitQuiet(IgniteUtils.java:7201)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.MarshallerContextImpl.className(MarshallerContextImpl.java:143)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.MarshallerContextAdapter.getClass(MarshallerContextAdapter.java:174)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(OptimizedMarshallerUtils.java:257)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride(OptimizedObjectInputStream.java:309)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.io.ObjectInputStream.readObject(ObjectInputStream.java:364)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.marshaller.optimized.OptimizedMarshaller.unmarshal(OptimizedMarshaller.java:248)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheMessage.unmarshalCollection(GridCacheMessage.java:599)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateRequest.finishUnmarshal(GridNearAtomicUpdateRequest.java:584)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheIoManager.unmarshall(GridCacheIoManager.java:996)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:268)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:197)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:76)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.cache.GridCacheIoManager$1$1$1.run(GridCacheIoManager.java:150)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6427)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:788)
      [11:51:56] : [org.apache.ignite:ignite-core] at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      [11:51:56] : [org.apache.ignite:ignite-core] at java.lang.Thread.run(Thread.java:745)

        Attachments

          Activity

            People

            • Assignee:
              avinogradov Anton Vinogradov
              Reporter:
              avinogradov Anton Vinogradov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: