... [2018-05-12 16:06:31,641][WARN ][tcp-comm-worker-#769%service.GridServiceProcessorMultiNodeConfigSelfTest6%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45015, failureDetectionTimeout=10000] [2018-05-12 16:06:31,641][WARN ][tcp-comm-worker-#769%service.GridServiceProcessorMultiNodeConfigSelfTest6%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45015]] [2018-05-12 16:06:31,641][ERROR][tcp-client-disco-sock-writer-#771%service.GridServiceProcessorMultiNodeConfigSelfTest6%][TcpDiscoverySpi] Failed to send message: TcpDiscoveryClientPingRequest [nodeToPing=2ca81dcb-b818-4772-8179-e0a9ff100005, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=483b6745361-ed0ae209-421b-4a90-8c06-99794fc00006, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]] java.net.SocketException: Socket is closed at java.net.Socket.getSendBufferSize(Socket.java:1215) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.socketStream(TcpDiscoverySpi.java:1370) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1482) at org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1256) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) [2018-05-12 16:06:31,642][INFO ][test-runner-#5440%service.GridServiceProcessorMultiNodeConfigSelfTest%][GridServiceProcessorMultiNodeConfigSelfTest5] >>> +-----------------------------------------------------+ >>> Ignite ver. 2.4.0-SNAPSHOT#19700101-sha1:DEV stopped OK >>> +-----------------------------------------------------+ >>> Ignite instance name: service.GridServiceProcessorMultiNodeConfigSelfTest5 >>> Grid uptime: 00:00:01.950 ... Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/192.168.43.79:45016, failureDetectionTimeout=10000] [2018-05-12 16:06:36,661][WARN ][tcp-comm-worker-#6%service.GridServiceProcessorMultiNodeConfigSelfTest1%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45016, /192.168.3.235:45016, /192.168.43.79:45016]] [2018-05-12 16:06:36,662][INFO ][tcp-comm-worker-#1%service.GridServiceProcessorMultiNodeConfigSelfTest0%][TcpDiscoverySpi] Pinging node: ed0ae209-421b-4a90-8c06-99794fc00006 [2018-05-12 16:06:36,662][INFO ][tcp-comm-worker-#1%service.GridServiceProcessorMultiNodeConfigSelfTest0%][TcpDiscoverySpi] Finished node ping [nodeId=ed0ae209-421b-4a90-8c06-99794fc00006, res=false, time=0ms] [2018-05-12 16:06:36,663][INFO ][test-runner-#5440%service.GridServiceProcessorMultiNodeConfigSelfTest%][GridServiceProcessorMultiNodeConfigSelfTest6] >>> +-----------------------------------------------------+ >>> Ignite ver. 2.4.0-SNAPSHOT#19700101-sha1:DEV stopped OK >>> +-----------------------------------------------------+ >>> Ignite instance name: service.GridServiceProcessorMultiNodeConfigSelfTest6 >>> Grid uptime: 00:00:06.117 [2018-05-12 16:07:50,949][ERROR][utility-#96%service.GridServiceProcessorMultiNodeConfigSelfTest1%][query] Failed to send event notification to node: ed0ae209-421b-4a90-8c06-99794fc00006 class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=ed0ae209-421b-4a90-8c06-99794fc00006, addrs=[127.0.0.1, 192.168.3.235, 192.168.43.79], sockAddrs=[/127.0.0.1:0, /192.168.3.235:0, /192.168.43.79:0], discPort=0, order=175, intOrder=91, lastExchangeTime=1526130389916, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=true], topic=T4 [topic=TOPIC_CACHE, id1=5dc785b4-dd3d-3c3b-b270-c5fe2d7ed9a2, id2=ed0ae209-421b-4a90-8c06-99794fc00006, id3=1], msg=GridContinuousMessage [type=MSG_EVT_NOTIFICATION, routineId=3df6e9bc-045e-4aa3-93bd-2181879e34ef, data=null, futId=null], policy=2] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1650) at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1862) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1397) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1368) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1350) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendNotification(GridContinuousProcessor.java:939) at org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.addNotification(GridContinuousProcessor.java:879) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:855) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.access$700(CacheContinuousQueryHandler.java:82) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler$1.onEntryUpdated(CacheContinuousQueryHandler.java:413) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.onEntryUpdated(CacheContinuousQueryManager.java:373) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1079) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userCommit(IgniteTxLocalAdapter.java:654) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.localFinish(GridDhtTxLocalAdapter.java:768) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.localFinish(GridDhtTxLocal.java:553) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.finishTx(GridDhtTxLocal.java:439) at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.commitDhtLocalAsync(GridDhtTxLocal.java:475) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:943) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:822) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:778) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:97) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:189) at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:187) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=ed0ae209-421b-4a90-8c06-99794fc00006, addrs=[127.0.0.1, 192.168.3.235, 192.168.43.79], sockAddrs=[/127.0.0.1:0, /192.168.3.235:0, /192.168.43.79:0], discPort=0, order=175, intOrder=91, lastExchangeTime=1526130389916, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=true] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2639) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2575) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1642) ... 35 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=ed0ae209-421b-4a90-8c06-99794fc00006, addrs=[/127.0.0.1:45016, /192.168.3.235:45016, /192.168.43.79:45016]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3313) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2863) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2750) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2611) ... 37 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45016, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3316) ... 40 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3153) ... 40 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/192.168.3.235:45016, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3316) ... 40 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3153) ... 40 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/192.168.43.79:45016, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3316) ... 40 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3153) ... 40 more [2018-05-12 16:08:16,176][WARN ][tcp-disco-msg-worker-#16%service.GridServiceProcessorMultiNodeConfigSelfTest2%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=edc139b9-def2-41b3-ad6a-20d5dd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1526130470978, loc=true, ver=2.4.0#19700101-sha1:00000000, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=af5b6745361-edc139b9-def2-41b3-ad6a-20d5dd900002, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=8b018527-1308-43e8-91a8-f3dae9300003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47503], discPort=47503, order=4, intOrder=4, lastExchangeTime=1526130307795, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=edc139b9-def2-41b3-ad6a-20d5dd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1526130470978, loc=true, ver=2.4.0#19700101-sha1:00000000, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=af5b6745361-edc139b9-def2-41b3-ad6a-20d5dd900002, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=8b018527-1308-43e8-91a8-f3dae9300003, order=4, addr=[127.0.0.1], daemon=false]]] [2018-05-12 16:08:16,178][WARN ][tcp-disco-msg-worker-#8%service.GridServiceProcessorMultiNodeConfigSelfTest1%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=ae1b8d9c-75ae-4635-8a17-81903dc00001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1526130470978, loc=true, ver=2.4.0#19700101-sha1:00000000, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=8f5b6745361-ae1b8d9c-75ae-4635-8a17-81903dc00001, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=edc139b9-def2-41b3-ad6a-20d5dd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1526130307310, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=ae1b8d9c-75ae-4635-8a17-81903dc00001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1526130470978, loc=true, ver=2.4.0#19700101-sha1:00000000, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=8f5b6745361-ae1b8d9c-75ae-4635-8a17-81903dc00001, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=edc139b9-def2-41b3-ad6a-20d5dd900002, order=3, addr=[127.0.0.1], daemon=false]]] [2018-05-12 16:08:16,207][WARN ][disco-event-worker-#36%service.GridServiceProcessorMultiNodeConfigSelfTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ed0ae209-421b-4a90-8c06-99794fc00006, addrs=[127.0.0.1, 192.168.3.235, 192.168.43.79], sockAddrs=[/127.0.0.1:0, /192.168.3.235:0, /192.168.43.79:0], discPort=0, order=175, intOrder=91, lastExchangeTime=1526130389916, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=true] [2018-05-12 16:09:10,540][WARN ][disco-event-worker-#84%service.GridServiceProcessorMultiNodeConfigSelfTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ed0ae209-421b-4a90-8c06-99794fc00006, addrs=[127.0.0.1, 192.168.3.235, 192.168.43.79], sockAddrs=[/127.0.0.1:0, /192.168.3.235:0, /192.168.43.79:0], discPort=0, order=175, intOrder=91, lastExchangeTime=1526130389916, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=true] [2018-05-12 16:09:10,548][WARN ][disco-event-worker-#84%service.GridServiceProcessorMultiNodeConfigSelfTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=edc139b9-def2-41b3-ad6a-20d5dd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1526130307310, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false] [2018-05-12 16:09:10,565][WARN ][disco-event-worker-#84%service.GridServiceProcessorMultiNodeConfigSelfTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=61cafed8-dc45-4723-b611-e655d7900000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1526130306840, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false] [2018-05-12 16:09:10,584][WARN ][disco-event-worker-#36%service.GridServiceProcessorMultiNodeConfigSelfTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ae1b8d9c-75ae-4635-8a17-81903dc00001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1526130306810, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false] [2018-05-12 16:09:10,585][WARN ][disco-event-worker-#84%service.GridServiceProcessorMultiNodeConfigSelfTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=8b018527-1308-43e8-91a8-f3dae9300003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47503], discPort=47503, order=4, intOrder=4, lastExchangeTime=1526130307785, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false] Node FAILED: TcpDiscoveryNode [id=8b018527-1308-43e8-91a8-f3dae9300003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47503], discPort=47503, order=4, intOrder=4, lastExchangeTime=1526130307785, loc=false, ver=2.4.0#19700101-sha1:00000000, isClient=false] Close incoming connection, unknown node [nodeId=8b018527-1308-43e8-91a8-f3dae9300003, ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=738058, bytesSent=511108, bytesRcvd0=42, bytesSent0=18, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=service.GridServiceProcessorMultiNodeConfigSelfTest1, finished=false, hashCode=20046412, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#74%service.GridServiceProcessorMultiNodeConfigSelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=38 lim=38 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:59126, createTime=1526130550580, closeTime=0, bytesSent=18, bytesRcvd=42, bytesSent0=18, bytesRcvd0=42, sndSchedTime=1526130550580, lastSndTime=1526130550580, lastRcvTime=1526130550580, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@1ef68d2f, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]]