[IGNITE-19925] NodeStoppingException upon stopping an embedded Ignite node

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Component/s: persistence
• Labels: ignite-3
• Assignee: Aleksandr Polovtsev
• Reporter: Ivan Artiukhov
• Created: 06/Jul/23 14:10
• Updated: 04/Sep/24 16:10
• Resolved: 06/Nov/23 11:16

Description

See the attached reproducer.

Steps:

• Start an embedded Ignite node.
• Create a table via key-vaue: 11 columns of type VARCHAR
• Insert 10 sample rows
• Stop the node via IgnitionManager#stop

Expected result:
No exceptions in the node's log

Actual result:
The following exception is seen:

Caused by: org.apache.ignite.lang.NodeStoppingException: IGN-CMN-1 TraceId:65d933f8-94bd-41e6-928d-7defcf52744c Operation has been cancelled (node is stopping).
        at org.apache.ignite.network.DefaultMessagingService.invoke0(DefaultMessagingService.java:227)
        at org.apache.ignite.network.DefaultMessagingService.invoke(DefaultMessagingService.java:159)
        at org.apache.ignite.network.MessagingService.invoke(MessagingService.java:145)
        at org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.sendWithRetry(TopologyAwareRaftGroupService.java:211)
        at org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.sendSubscribeMessage(TopologyAwareRaftGroupService.java:197)
        at org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService.unsubscribeLeader(TopologyAwareRaftGroupService.java:329)
        at org.apache.ignite.internal.replicator.Replica.shutdown(Replica.java:278)
        at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
        at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
        at org.apache.ignite.internal.replicator.ReplicaManager.stopReplicaInternal(ReplicaManager.java:410)
        at org.apache.ignite.internal.replicator.ReplicaManager.stopReplica(ReplicaManager.java:385)
        at org.apache.ignite.internal.table.distributed.TableManager.lambda$cleanUpTablesResources$30(TableManager.java:1093)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at org.apache.ignite.internal.table.distributed.TableManager.cleanUpTablesResources(TableManager.java:1119)
        at org.apache.ignite.internal.table.distributed.TableManager.stop(TableManager.java:1045)
        at org.apache.ignite.internal.app.LifecycleManager.lambda$stopAllComponents$1(LifecycleManager.java:133)
        at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
        at org.apache.ignite.internal.app.LifecycleManager.stopAllComponents(LifecycleManager.java:131)
        at org.apache.ignite.internal.app.LifecycleManager.stopNode(LifecycleManager.java:115)
        at org.apache.ignite.internal.app.IgniteImpl.stop(IgniteImpl.java:807)
        at org.apache.ignite.internal.app.IgnitionImpl.lambda$stop$0(IgnitionImpl.java:109)
        at java.base/java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1822)
        at org.apache.ignite.internal.app.IgnitionImpl.stop(IgnitionImpl.java:108)
        at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:96)
        at org.apache.ignite.IgnitionManager.stop(IgnitionManager.java:82)
        at org.apache.ignite.example.AbstractExamplesTest.stopNode(AbstractExamplesTest.java:76)

git bisect says that the following commit introduced the bug (belongs to IGNITE-19199):

b6004047b3c3e9cd91b5ccf28c26ee206c1e3a7f is the first bad commit

Issue Links

• is caused by: IGNITE-19199 Idle safe time propagation for the metastorage
• links to: GitHub Pull Request #2573
• links to: GitHub Pull Request #2792