[IGNITE-9206] Node can't join to ring if all existing nodes have stopped and another new node joined ahead

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.5
• Component/s: cache
• Assignee: Unassigned
• Reporter: Pavel Kovalenko
• Created: 07/Aug/18 11:05
• Updated: 26/Jun/20 18:49

Description

TcpDiscovery SPI problem.
Situation:
Existing cluster with nodes 1 and 2. Nodes 1 and 2 are stopping.
1) Node 3 joins to cluster and sends JoinMessage to node 2.
2) Node 2 is stopping and unable to handle JoinMessage from node 3. Node 3 choose node 1 as the next node to send the message.
3) Node 3 sends JoinMessage to node 1.
4) Node 4 joins to cluster.
5) Node 1 is stopping and unable to handle JoinMessage from node 3.
6) Node 4 sees that there are no alive nodes in the ring at the time and become the first node in the topology.
7) Node 3 sends JoinMessage to Node 4 and this process repeats again and again without any success.
At Node 4 logs we can see that remote connection from Node 3 is established but no active actions have performed. Node 3 leaves in CONNECTING state forever. At the same time Node 4 thinks that Node 3 is already in the ring.

Failed test:
GridCacheReplicatedDataStructuresFailoverSelfTest#testAtomicSequenceConstantTopologyChange

Link to TC:
https://ci.ignite.apache.org/viewLog.html?buildId=1594376&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_DataStructures

Shrinked log:

[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,733][INFO ][main][root] >>> Stopping grid [name=replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0, id=3e2c94bd-8e98-4dd9-8d1a-befbfe000000]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,739][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][root] Start node: replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,740][INFO ][tcp-disco-msg-worker-#2146%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3e2c94bd-8e98-4dd9-8d1a-befbfe000000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1533416953738, loc=false, ver=2.7.0#20180803-sha1:3ab8bbad, isClient=false]]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,741][INFO ][tcp-disco-srvr-#2100%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=50099]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,741][INFO ][tcp-disco-srvr-#2100%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=50099]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,743][INFO ][tcp-disco-sock-reader-#2151%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:50099, rmtPort=50099]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,746][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] 
[00:09:13] :	 [Step 3/4] 
[00:09:13] :	 [Step 3/4] >>>    __________  ________________  
[00:09:13] :	 [Step 3/4] >>>   /  _/ ___/ |/ /  _/_  __/ __/  
[00:09:13] :	 [Step 3/4] >>>  _/ // (7 7    // /  / / / _/    
[00:09:13] :	 [Step 3/4] >>> /___/\___/_/|_/___/ /_/ /___/   
[00:09:13] :	 [Step 3/4] >>> 
[00:09:13] :	 [Step 3/4] >>> ver. 2.7.0-SNAPSHOT#20180803-sha1:3ab8bbad
[00:09:13] :	 [Step 3/4] >>> 2018 Copyright(C) Apache Software Foundation
[00:09:13] :	 [Step 3/4] >>> 
[00:09:13] :	 [Step 3/4] >>> Ignite documentation: http://ignite.apache.org
[00:09:13] :	 [Step 3/4] 
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,746][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] Config URL: n/a
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,747][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] IgniteConfiguration [igniteInstanceName=replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/9198da4c51c3e112, igniteWorkDir=/data/teamcity/work/9198da4c51c3e112/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@13fed1ec, nodeId=fe9e7ca7-c0fa-4b51-8a87-1255f8c00007, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@514987e3, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45170, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@65788f9a[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4fb8a27b, addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=REPLICATED, backups=0, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268895436, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@a63cd72, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], authEnabled=false, failureHnd=NoOpFailureHandler [], commFailureRslvr=null]
...
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,750][INFO ][tcp-disco-sock-reader-#2151%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:50099, rmtPort=50099
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,750][INFO ][tcp-disco-sock-reader-#2145%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39170, rmtPort=39170
[00:09:13]W:		 [org.apache.ignite:ignite-core] [2018-08-04 21:09:13,842][WARN ][tcp-disco-msg-worker-#2146%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi'
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,749][INFO ][tcp-disco-sock-reader-#2139%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:33299, rmtPort=33299
...
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,875][INFO ][main][GridCacheReplicatedDataStructuresFailoverSelfTest0] 
[00:09:13] :	 [Step 3/4] 
[00:09:13] :	 [Step 3/4] >>> +------------------------------------------------------------------------------------------+
[00:09:13] :	 [Step 3/4] >>> Ignite ver. 2.7.0-SNAPSHOT#20180803-sha1:3ab8bbadc80446e53261da784bfbed48b6ab5322 stopped OK
[00:09:13] :	 [Step 3/4] >>> +------------------------------------------------------------------------------------------+
[00:09:13] :	 [Step 3/4] >>> Ignite instance name: replicated.GridCacheReplicatedDataStructuresFailoverSelfTest0
[00:09:13] :	 [Step 3/4] >>> Grid uptime: 00:00:02.811
[00:09:13] :	 [Step 3/4] 
[00:09:13] :	 [Step 3/4] 
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,876][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45170, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false]
[00:09:13]W:		 [org.apache.ignite:ignite-core] [2018-08-04 21:09:13,876][WARN ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
[00:09:13]W:		 [org.apache.ignite:ignite-core] [2018-08-04 21:09:13,903][WARN ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival).
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,903][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] Security status [authentication=off, tls/ssl=off]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,912][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][ClientListenerProcessor] Client connector processor has started on TCP port 10800
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,916][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] Non-loopback local IPs: 172.17.0.6
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,916][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][GridCacheReplicatedDataStructuresFailoverSelfTest7] Enabled local MACs: 0242AC110006
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,917][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpDiscoverySpi] Connection check threshold is calculated: 10000
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,918][INFO ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=/127.0.0.1, locNodeId=fe9e7ca7-c0fa-4b51-8a87-1255f8c00007]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,926][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=35863]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,926][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=35863]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,926][INFO ][tcp-disco-sock-reader-#2156%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:35863, rmtPort=35863]
[00:09:13] :	 [Step 3/4] [2018-08-04 21:09:13,933][INFO ][tcp-disco-sock-reader-#2156%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:35863, rmtPort=35863
[00:09:15] :	 [Step 3/4] [2018-08-04 21:09:15,938][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=38642]
[00:09:15] :	 [Step 3/4] [2018-08-04 21:09:15,938][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=38642]
[00:09:15] :	 [Step 3/4] [2018-08-04 21:09:15,939][INFO ][tcp-disco-sock-reader-#2157%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:38642, rmtPort=38642]
[00:09:15] :	 [Step 3/4] [2018-08-04 21:09:15,942][INFO ][tcp-disco-sock-reader-#2157%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:38642, rmtPort=38642
[00:09:17] :	 [Step 3/4] [2018-08-04 21:09:17,945][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=41001]
[00:09:17] :	 [Step 3/4] [2018-08-04 21:09:17,945][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=41001]

[00:09:20] :	 [Step 3/4] [2018-08-04 21:09:20,744][INFO ][tcp-disco-msg-worker-#2154%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=79ab4ba3-98df-488b-95b6-68e0e6a00006, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, intOrder=2, lastExchangeTime=1533416960734, loc=false, ver=2.7.0#20180803-sha1:3ab8bbad, isClient=false]]
[00:09:20] :	 [Step 3/4] [2018-08-04 21:09:20,745][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60760]
[00:09:20] :	 [Step 3/4] [2018-08-04 21:09:20,745][INFO ][tcp-disco-srvr-#2147%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60760]
[00:09:20] :	 [Step 3/4] [2018-08-04 21:09:20,745][INFO ][tcp-disco-sock-reader-#2163%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60760, rmtPort=60760]
[00:09:25] :	 [Step 3/4] [2018-08-04 21:09:25,745][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=58381]
[00:09:25] :	 [Step 3/4] [2018-08-04 21:09:25,746][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=58381]
...
[00:14:16] :	 [Step 3/4] [2018-08-04 21:14:16,143][INFO ][tcp-disco-sock-reader-#2222%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39278, rmtPort=39278]
[00:14:16] :	 [Step 3/4] [2018-08-04 21:14:16,145][INFO ][tcp-disco-sock-reader-#2222%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39278, rmtPort=39278
[00:14:21]W:		 [org.apache.ignite:ignite-core] [2018-08-04 21:14:21,145][WARN ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6][TcpDiscoverySpi] Node has not been connected to topology and will repeat join process. Check remote nodes logs for possible error messages. Note that large topology may require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' configuration property if getting this message on the starting nodes [networkTimeout=5000]
[00:14:21] :	 [Step 3/4] [2018-08-04 21:14:21,146][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=48489]
[00:14:21] :	 [Step 3/4] [2018-08-04 21:14:21,147][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=48489]
...
[00:19:16] :	 [Step 3/4] [2018-08-04 21:19:16,639][INFO ][tcp-disco-sock-reader-#2282%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39305, rmtPort=39305]
[00:19:16] :	 [Step 3/4] [2018-08-04 21:19:16,643][INFO ][tcp-disco-sock-reader-#2282%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39305, rmtPort=39305
[00:19:21]W:		 [org.apache.ignite:ignite-core] [2018-08-04 21:19:21,645][WARN ][thread-replicated.GridCacheReplicatedDataStructuresFailoverSelfTest6][TcpDiscoverySpi] Node has not been connected to topology and will repeat join process. Check remote nodes logs for possible error messages. Note that large topology may require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' configuration property if getting this message on the starting nodes [networkTimeout=5000]
[00:19:21] :	 [Step 3/4] [2018-08-04 21:19:21,646][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=41240]
[00:19:21] :	 [Step 3/4] [2018-08-04 21:19:21,647][INFO ][tcp-disco-srvr-#2155%replicated.GridCacheReplicatedDataStructuresFailoverSelfTest7%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=41240]
...

Issue Links

• relates to: IGNITE-8785 Node may hang indefinitely in CONNECTING state during cluster segmentation