Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-20012

Raft client freezing on stop

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • Docs Required, Release Notes Required

    Description

      The client cannot be closed while a RAFT command is being handled. The request cannot be processed due to the infinity of waiting for schema.

      "Test worker" #1 prio=5 os_prio=0 cpu=84323.53ms elapsed=3507.65s tid=0x00007f224402d000 nid=0x153c93 waiting on condition  [0x00007f22493f0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(java.base@11.0.17/Native Method)
        at org.apache.ignite.internal.util.IgniteSpinReadWriteLock.writeLock(IgniteSpinReadWriteLock.java:255)
        at org.apache.ignite.internal.util.IgniteSpinBusyLock.block(IgniteSpinBusyLock.java:68)
        at org.apache.ignite.internal.raft.RaftGroupServiceImpl.shutdown(RaftGroupServiceImpl.java:463)
        at org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.shutdown(TopologyAwareRaftGroupService.java:430)
        at org.apache.ignite.internal.table.distributed.storage.InternalTableImpl$$Lambda$3966/0x0000000801135040.close(Unknown Source)
        at org.apache.ignite.internal.util.IgniteUtils.lambda$closeAllManually$1(IgniteUtils.java:685)
        at org.apache.ignite.internal.util.IgniteUtils$$Lambda$3937/0x000000080112c440.accept(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@11.0.17/ForEachOps.java:183)
        at java.util.stream.ReferencePipeline$2$1.accept(java.base@11.0.17/ReferencePipeline.java:177)
        at java.util.stream.StreamSpliterators$WrappingSpliterator$$Lambda$38/0x000000080008b040.accept(java.base@11.0.17/Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(java.base@11.0.17/ReferencePipeline.java:195)
        at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$ValueSpliterator.acceptOnIndex(Int2ObjectOpenHashMap.java:1221)
        at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$ValueSpliterator.acceptOnIndex(Int2ObjectOpenHashMap.java:1205)
        at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$MapSpliterator.forEachRemaining(Int2ObjectOpenHashMap.java:876)
        at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap$ValueSpliterator.forEachRemaining(Int2ObjectOpenHashMap.java:1205)
        at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.17/AbstractPipeline.java:484)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.17/AbstractPipeline.java:474)
        at java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(java.base@11.0.17/StreamSpliterators.java:312)
        at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(java.base@11.0.17/Streams.java:734)
        at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.17/AbstractPipeline.java:484)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.17/AbstractPipeline.java:474)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@11.0.17/ForEachOps.java:150)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(java.base@11.0.17/ForEachOps.java:173)
        at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.17/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(java.base@11.0.17/ReferencePipeline.java:497)
        at org.apache.ignite.internal.util.IgniteUtils.closeAllManually(IgniteUtils.java:683)
        at org.apache.ignite.internal.table.distributed.storage.InternalTableImpl.close(InternalTableImpl.java:1649)
        at org.apache.ignite.internal.util.IgniteUtils.lambda$closeAllManually$1(IgniteUtils.java:685)
        at org.apache.ignite.internal.util.IgniteUtils$$Lambda$3937/0x000000080112c440.accept(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(java.base@11.0.17/ForEachOps.java:183)
        at java.util.stream.ReferencePipeline$2$1.accept(java.base@11.0.17/ReferencePipeline.java:177)
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(java.base@11.0.17/Spliterators.java:948)
        at java.util.stream.AbstractPipeline.copyInto(java.base@11.0.17/AbstractPipeline.java:484)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.17/AbstractPipeline.java:474)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(java.base@11.0.17/ForEachOps.java:150)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(java.base@11.0.17/ForEachOps.java:173)
        at java.util.stream.AbstractPipeline.evaluate(java.base@11.0.17/AbstractPipeline.java:234)
        at java.util.stream.ReferencePipeline.forEach(java.base@11.0.17/ReferencePipeline.java:497)
        at org.apache.ignite.internal.util.IgniteUtils.closeAllManually(IgniteUtils.java:683)
        at org.apache.ignite.internal.util.IgniteUtils.closeAllManually(IgniteUtils.java:706)
        at org.apache.ignite.internal.table.distributed.TableManager.cleanUpTablesResources(TableManager.java:1397)
        at org.apache.ignite.internal.table.distributed.TableManager.stop(TableManager.java:1320)
        at org.apache.ignite.internal.app.LifecycleManager.lambda$stopAllComponents$1(LifecycleManager.java:133)
        at org.apache.ignite.internal.app.LifecycleManager$$Lambda$3900/0x000000080111a440.accept(Unknown Source)
        at java.util.Iterator.forEachRemaining(java.base@11.0.17/Iterator.java:133)
        at org.apache.ignite.internal.app.LifecycleManager.stopAllComponents(LifecycleManager.java:131)
        - locked <0x0000000415da2360> (a org.apache.ignite.internal.app.LifecycleManager)
        at org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:115)
        at org.apache.ignite.internal.app.IgniteImpl.stop(IgniteImpl.java:847)
        at org.apache.ignite.internal.app.IgnitionImpl.lambda$stop$0(IgnitionImpl.java:109)
        at org.apache.ignite.internal.app.IgnitionImpl$$Lambda$3856/0x000000080110ec40.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.computeIfPresent(java.base@11.0.17/ConcurrentHashMap.java:1822)
        - locked <0x000000043a838ed0> (a java.util.concurrent.ConcurrentHashMap$Node)
        at org.apache.ignite.internal.app.IgnitionImpl.stop(IgnitionImpl.java:108)
        at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:96)
        at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:82)
        at org.apache.ignite.internal.Cluster.lambda$shutdown$11(Cluster.java:429)
        at org.apache.ignite.internal.Cluster$$Lambda$4236/0x0000000801312040.accept(Unknown Source)
        at java.util.ArrayList.forEach(java.base@11.0.17/ArrayList.java:1541)
        at org.apache.ignite.internal.Cluster.shutdown(Cluster.java:429)
        at org.apache.ignite.internal.raftsnapshot.ItTableRaftSnapshotsTest.shutdownCluster(ItTableRaftSnapshotsTest.java:151)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.17/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.17/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.17/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.17/Method.java:566)
        at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)

"%node1%JRaft-Request-Processor-37" #113906 daemon prio=10 os_prio=0 cpu=3.06ms elapsed=395.00s tid=0x00007f889410d000 nid=0xf4135 waiting on condition  [0x00007f81b87c5000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.17/Native Method)
	- parking to wait for  <0x0000000408a88488> (a java.util.concurrent.CompletableFuture$Signaller)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.17/LockSupport.java:194)
	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@11.0.17/CompletableFuture.java:1796)
	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@11.0.17/ForkJoinPool.java:3128)
	at java.util.concurrent.CompletableFuture.waitingGet(java.base@11.0.17/CompletableFuture.java:1823)
	at java.util.concurrent.CompletableFuture.join(java.base@11.0.17/CompletableFuture.java:2043)
	at org.apache.ignite.internal.schema.registry.SchemaRegistryImpl.schema(SchemaRegistryImpl.java:90)
	at org.apache.ignite.internal.schema.registry.SchemaRegistryImpl.waitLatestSchema(SchemaRegistryImpl.java:128)
	at org.apache.ignite.internal.index.IndexManager$TableRowToIndexKeyConverter.createConverter(IndexManager.java:545)
	at org.apache.ignite.internal.index.IndexManager$TableRowToIndexKeyConverter.convert(IndexManager.java:530)
	- locked <0x0000000408a88540> (a java.lang.Object)
	at org.apache.ignite.internal.index.IndexManager$$Lambda$2740/0x0000000800d83c40.apply(Unknown Source)
	at org.apache.ignite.internal.table.distributed.HashIndexLocker.locksForLookup(HashIndexLocker.java:65)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.resolveRowByPk(PartitionReplicaListener.java:1333)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processSingleEntryAction(PartitionReplicaListener.java:1882)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$processRequest$1(PartitionReplicaListener.java:350)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener$$Lambda$3801/0x000000080109d440.get(Unknown Source)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.appendTxCommand(PartitionReplicaListener.java:1406)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.processRequest(PartitionReplicaListener.java:350)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.lambda$invoke$0(PartitionReplicaListener.java:343)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener$$Lambda$2559/0x0000000800d21040.apply(Unknown Source)
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(java.base@11.0.17/CompletableFuture.java:1072)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.17/CompletableFuture.java:506)
	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.17/CompletableFuture.java:2073)
	at org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$sendWithRetry$39(RaftGroupServiceImpl.java:541)
	at org.apache.ignite.internal.raft.RaftGroupServiceImpl$$Lambda$1443/0x00000008009a8c40.accept(Unknown Source)
	at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.17/CompletableFuture.java:859)
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(java.base@11.0.17/CompletableFuture.java:837)
	at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.17/CompletableFuture.java:506)
	at java.util.concurrent.CompletableFuture.complete(java.base@11.0.17/CompletableFuture.java:2073)
	at org.apache.ignite.network.DefaultMessagingService.onInvokeResponse(DefaultMessagingService.java:401)
	at org.apache.ignite.network.DefaultMessagingService.send0(DefaultMessagingService.java:200)
	at org.apache.ignite.network.DefaultMessagingService.respond(DefaultMessagingService.java:143)
	at org.apache.ignite.network.MessagingService.respond(MessagingService.java:89)
	at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$NetworkRpcContext.sendResponse(IgniteRpcServer.java:233)
	at org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:52)
	at org.apache.ignite.raft.jraft.rpc.RpcRequestProcessor.handleRequest(RpcRequestProcessor.java:29)
	at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler.lambda$onReceived$0(IgniteRpcServer.java:192)
	at org.apache.ignite.raft.jraft.rpc.impl.IgniteRpcServer$RpcMessageHandler$$Lambda$1442/0x00000008009a8440.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.17/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.17/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.17/Thread.java:834)

       

      Attachments

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. IGNITE-19227 Wait for schema availability outside JRaft threads

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IGNITE-19227 Wait for schema availability outside JRaft threads

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              v.pyatkov Vladislav Pyatkov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: