[IGNITE-12398] Apache Ignite Cluster(Amazon S3 Based Discovery) Nodes getting down if we connect Ignite Visor Command Line Interface

• Type: Bug
• Status: In Progress
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.7
• Component/s: aws, general, s3, visor

Environment

Production

• Flags: Patch, Important
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Emmanouil Gkatziouras
• Reporter: Ravi Kumar Powli
• Reviewer: Dmitriy Sorokin
• Created: 27/Nov/19 11:16
• Updated: 29/Dec/20 08:56

Description

We have Apache Ignite 3 node cluster setup with Amazon S3 Based Discovery. If we connect any one cluster node using Ignite Visor Command Line Interface it got hang and automatically cluster nodes(all the three nodes) are going down. Please find the below exception stacktrace.

[SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.NullPointerException]]
java.lang.NullPointerException
at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.key(TcpDiscoveryS3IpFinder.java:247)
at org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.registerAddresses(TcpDiscoveryS3IpFinder.java:205)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddFinishedMessage(ServerImpl.java:4616)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4232)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2816)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2611)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7188)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[10:36:54,600][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=DataStoreIgniteCache, finished=true, heartbeatTs=1574332614423]]]
class org.apache.ignite.IgniteException: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=DataStoreIgniteCache, finished=true, heartbeatTs=1574332614423]
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at org.apache.ignite.internal.worker.WorkersRegistry.onStopped(WorkersRegistry.java:169)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:153)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
[10:36:59] Ignite node stopped OK [name=DataStoreIgniteCache, uptime=00:01:13.934]