[17:36:13,306][INFO ][main][root] >>> Starting test: TcpDiscoverySelfTest#testFailedNodes4 <<< [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> 2017 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Config URL: n/a [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Daemon mode: off [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] OS: Mac OS X 10.12.4 x86_64 [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] OS user: yzhdanov [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] PID: 7858 [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Language runtime: Java Platform API Specification ver. 1.8 [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] VM information: Java(TM) SE Runtime Environment 1.8.0-b132 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.0-b70 [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] VM total memory: 2.9GB [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Remote Management [restart: off, REST: off, JMX (remote: off)] [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] IGNITE_HOME=/Users/yzhdanov/projects/incubator-ignite [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] VM arguments: [-ea, -Xmx3g, -Xms3g, -Didea.launcher.port=7533, -Didea.launcher.bin.path=/Applications/IntelliJ IDEA CE.app/Contents/bin, -Dfile.encoding=UTF-8] [17:36:13,312][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Configured caches ['ignite-sys-cache', 'ignite-atomics-sys-cache'] [17:36:13,313][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] 3-rd party licenses can be found at: /Users/yzhdanov/projects/incubator-ignite/libs/licenses [17:36:13,312][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [17:36:13,316][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] Configured plugins: [17:36:13,316][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- TestPlatformPlugin 1.0 [17:36:13,316][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- - [17:36:13,316][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] [17:36:13,318][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45010, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [17:36:13,319][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [17:36:13,324][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Security status [authentication=off, tls/ssl=off] [17:36:13,324][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [17:36:13,324][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi. [17:36:13,336][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Network timeout is too low (at least 3000 ms recommended): 2500 [17:36:13,335][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Non-loopback local IPs: 172.25.4.215, fe80:0:0:0:394f:9f9f:4d7e:efc%utun0, fe80:0:0:0:c53:b736:8469:7030%en1 [17:36:13,335][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Enabled local MACs: 1040F39C4BE4 [17:36:13,337][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Successfully bound to TCP port [port=47500, localHost=/127.0.0.1, locNodeId=1eb76302-be6c-4ad7-bf29-a6476d100000] [17:36:13,345][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-sys-cache, mode=REPLICATED] [17:36:13,345][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-atomics-sys-cache, mode=PARTITIONED] [17:36:13,348][INFO ][exchange-worker-#825%tcp.TcpDiscoverySelfTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, node=1eb76302-be6c-4ad7-bf29-a6476d100000] [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Performance suggestions for grid 'tcp.TcpDiscoverySelfTest0' (fix if possible) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Disable checkpoints (remove 'checkpointSpi' from configuration) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Disable grid events (remove 'includeEventTypes' from configuration) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Use default binary marshaller (do not set 'marshaller' explicitly) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] ^-- Disable assertions (remove '-ea' from JVM options) [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest0] >>> +------------------------------------------+ >>> Ignite ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> +------------------------------------------+ >>> OS name: Mac OS X 10.12.4 x86_64 >>> CPU(s): 8 >>> Heap: 3.0GB >>> VM name: 7858@mbp.local >>> Ignite instance name: tcp.TcpDiscoverySelfTest0 >>> Local node [ID=1EB76302-BE6C-4AD7-BF29-A6476D100000, order=1, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:45010 TCP:47500 [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridDiscoveryManager] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=3.0GB] [17:36:13,353][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][root] Node started with the following configuration [id=1eb76302-be6c-4ad7-bf29-a6476d100000, marshaller=org.apache.ignite.marshaller.optimized.OptimizedMarshaller@5eacd566, binaryCfg=null, lateAff=true] [17:36:13,358][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> 2017 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Config URL: n/a [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Daemon mode: off [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] OS: Mac OS X 10.12.4 x86_64 [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] OS user: yzhdanov [17:36:13,357][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] PID: 7858 [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Language runtime: Java Platform API Specification ver. 1.8 [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] VM information: Java(TM) SE Runtime Environment 1.8.0-b132 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.0-b70 [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] VM total memory: 2.9GB [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Remote Management [restart: off, REST: off, JMX (remote: off)] [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] IGNITE_HOME=/Users/yzhdanov/projects/incubator-ignite [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] VM arguments: [-ea, -Xmx3g, -Xms3g, -Didea.launcher.port=7533, -Didea.launcher.bin.path=/Applications/IntelliJ IDEA CE.app/Contents/bin, -Dfile.encoding=UTF-8] [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Configured caches ['ignite-sys-cache', 'ignite-atomics-sys-cache'] [17:36:13,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] 3-rd party licenses can be found at: /Users/yzhdanov/projects/incubator-ignite/libs/licenses [17:36:13,361][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] Configured plugins: [17:36:13,361][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- TestPlatformPlugin 1.0 [17:36:13,361][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- - [17:36:13,361][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] [17:36:13,364][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45011, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false][17:36:13,364][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [17:36:13,369][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [17:36:13,369][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi. [17:36:13,369][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Security status [authentication=off, tls/ssl=off] [17:36:13,375][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Non-loopback local IPs: 172.25.4.215, fe80:0:0:0:394f:9f9f:4d7e:efc%utun0, fe80:0:0:0:c53:b736:8469:7030%en1 [17:36:13,375][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Enabled local MACs: 1040F39C4BE4 [17:36:13,375][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Network timeout is too low (at least 3000 ms recommended): 2500 [17:36:13,377][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Successfully bound to TCP port [port=47501, localHost=/127.0.0.1, locNodeId=801bd00d-7408-4dc2-ac92-928b6a400001] [17:36:13,455][INFO ][disco-event-worker-#822%tcp.TcpDiscoverySelfTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967373381, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:13,455][INFO ][disco-event-worker-#822%tcp.TcpDiscoverySelfTest0%][GridDiscoveryManager] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=3.0GB] [17:36:13,458][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-sys-cache, mode=REPLICATED] [17:36:13,458][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-atomics-sys-cache, mode=PARTITIONED] [17:36:13,459][INFO ][grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51076] [17:36:13,459][INFO ][grid-nio-worker-tcp-comm-0-#816%tcp.TcpDiscoverySelfTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45010, rmtAddr=/127.0.0.1:51078] [17:36:13,461][INFO ][grid-nio-worker-tcp-comm-1-#817%tcp.TcpDiscoverySelfTest0%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:51076, rmtAddr=/127.0.0.1:45011] [17:36:13,669][INFO ][exchange-worker-#825%tcp.TcpDiscoverySelfTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, node=801bd00d-7408-4dc2-ac92-928b6a400001] [17:36:13,671][INFO ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, ignite-atomics-sys-cache]] [17:36:13,671][INFO ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Rebalancing started [top=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, node=801bd00d-7408-4dc2-ac92-928b6a400001] [17:36:13,671][INFO ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridDhtPartitionDemander] Starting rebalancing [mode=SYNC, fromNode=1eb76302-be6c-4ad7-bf29-a6476d100000, partitionsCount=100, topology=AffinityTopologyVersion [topVer=2, minorTopVer=0], updateSeq=1] [17:36:13,673][INFO ][utility-#856%tcp.TcpDiscoverySelfTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [fromNode=1eb76302-be6c-4ad7-bf29-a6476d100000, topology=AffinityTopologyVersion [topVer=2, minorTopVer=0], time=10 ms] [17:36:13,673][INFO ][utility-#856%tcp.TcpDiscoverySelfTest1%][GridDhtPartitionDemander] Starting rebalancing [mode=SYNC, fromNode=1eb76302-be6c-4ad7-bf29-a6476d100000, partitionsCount=500, topology=AffinityTopologyVersion [topVer=2, minorTopVer=0], updateSeq=1] [17:36:13,677][INFO ][sys-#858%tcp.TcpDiscoverySelfTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [fromNode=1eb76302-be6c-4ad7-bf29-a6476d100000, topology=AffinityTopologyVersion [topVer=2, minorTopVer=0], time=0 ms] [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Performance suggestions for grid 'tcp.TcpDiscoverySelfTest1' (fix if possible) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Disable checkpoints (remove 'checkpointSpi' from configuration) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Disable grid events (remove 'includeEventTypes' from configuration) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Use default binary marshaller (do not set 'marshaller' explicitly) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] ^-- Disable assertions (remove '-ea' from JVM options) [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest1] >>> +------------------------------------------+ >>> Ignite ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> +------------------------------------------+ >>> OS name: Mac OS X 10.12.4 x86_64 >>> CPU(s): 8 >>> Heap: 3.0GB >>> VM name: 7858@mbp.local >>> Ignite instance name: tcp.TcpDiscoverySelfTest1 >>> Local node [ID=801BD00D-7408-4DC2-AC92-928B6A400001, order=2, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:45011 TCP:47501 [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridDiscoveryManager] Topology snapshot [ver=2, servers=2, clients=0, CPUs=8, heap=3.0GB] [17:36:13,680][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][root] Node started with the following configuration [id=801bd00d-7408-4dc2-ac92-928b6a400001, marshaller=org.apache.ignite.marshaller.optimized.OptimizedMarshaller@4363aef9, binaryCfg=null, lateAff=true] [17:36:13,684][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> 2017 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Config URL: n/a [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Daemon mode: off [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] OS: Mac OS X 10.12.4 x86_64 [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] OS user: yzhdanov [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] PID: 7858 [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Language runtime: Java Platform API Specification ver. 1.8 [17:36:13,683][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] VM information: Java(TM) SE Runtime Environment 1.8.0-b132 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.0-b70 [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] VM total memory: 2.9GB [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Remote Management [restart: off, REST: off, JMX (remote: off)] [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] IGNITE_HOME=/Users/yzhdanov/projects/incubator-ignite [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] VM arguments: [-ea, -Xmx3g, -Xms3g, -Didea.launcher.port=7533, -Didea.launcher.bin.path=/Applications/IntelliJ IDEA CE.app/Contents/bin, -Dfile.encoding=UTF-8] [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Configured caches ['ignite-sys-cache', 'ignite-atomics-sys-cache'] [17:36:13,684][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] 3-rd party licenses can be found at: /Users/yzhdanov/projects/incubator-ignite/libs/licenses [17:36:13,686][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] Configured plugins: [17:36:13,686][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- TestPlatformPlugin 1.0 [17:36:13,686][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- - [17:36:13,686][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] [17:36:13,688][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45012, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [17:36:13,688][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [17:36:13,692][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [17:36:13,692][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi. [17:36:13,692][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Security status [authentication=off, tls/ssl=off] [17:36:13,696][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Non-loopback local IPs: 172.25.4.215, fe80:0:0:0:394f:9f9f:4d7e:efc%utun0, fe80:0:0:0:c53:b736:8469:7030%en1 [17:36:13,696][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Enabled local MACs: 1040F39C4BE4 [17:36:13,697][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Network timeout is too low (at least 3000 ms recommended): 2500 [17:36:13,698][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Successfully bound to TCP port [port=47502, localHost=/127.0.0.1, locNodeId=1db622e7-f722-4b86-a09e-d046cd900002] [17:36:13,710][INFO ][tcp-disco-msg-worker-#1139%tcp.TcpDiscoverySelfTest2%][TcpDiscoverySelfTest$TestFailedNodesSpi] IO error on message send [locNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=0, intOrder=3, lastExchangeTime=1490967373695, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], msg=TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=0, intOrder=3, lastExchangeTime=1490967373707, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], dataPacket=null, discardMsgId=null, discardCustomMsgId=null, top=null, clientTop=null, gridStartTime=1490967373338, super=TcpDiscoveryAbstractMessage [sndNodeId=801bd00d-7408-4dc2-ac92-928b6a400001, id=bdef3942b51-1eb76302-be6c-4ad7-bf29-a6476d100000, verifierNodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]] [17:36:13,713][INFO ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:13,713][INFO ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Topology snapshot [ver=3, servers=3, clients=0, CPUs=8, heap=3.0GB] [17:36:13,713][INFO ][tcp-disco-msg-worker-#1139%tcp.TcpDiscoverySelfTest2%][TcpDiscoverySelfTest$TestFailedNodesSpi] Skip messages send and stop node [locNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=0, intOrder=3, lastExchangeTime=1490967373707, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], msg=TcpDiscoveryNodeFailedMessage [failedNodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, order=1, warning=null, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=3fef3942b51-1db622e7-f722-4b86-a09e-d046cd900002, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=[1eb76302-be6c-4ad7-bf29-a6476d100000], isClient=false]]] [17:36:13,713][WARN ][tcp-disco-msg-worker-#1139%tcp.TcpDiscoverySelfTest2%][TcpDiscoverySelfTest$TestFailedNodesSpi] Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi' [17:36:16,205][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Node has not been connected to topology and will repeat join process. Check remote nodes logs for possible error messages. Note that large topology may require significant time to start. Increase 'TcpDiscoverySpi.networkTimeout' configuration property if getting this message on the starting nodes [networkTimeout=2500] [17:36:18,709][WARN ][tcp-disco-sock-reader-#1145%tcp.TcpDiscoverySelfTest0%][TcpDiscoverySelfTest$TestFailedNodesSpi] Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=2500] [17:36:18,709][WARN ][tcp-disco-sock-reader-#1145%tcp.TcpDiscoverySelfTest0%][TcpDiscoverySelfTest$TestFailedNodesSpi] Socket operation timed out on handshake (consider increasing 'networkTimeout' configuration property) [netTimeout=2500] [17:36:26,214][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=2488] [17:36:33,717][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=801bd00d-7408-4dc2-ac92-928b6a400001]. Dumping pending objects that might be the cause: [17:36:33,717][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Ready affinity version: AffinityTopologyVersion [topVer=2, minorTopVer=0] [17:36:33,720][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000, 1db622e7-f722-4b86-a09e-d046cd900002], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967393534, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:33,720][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending exchange futures: [17:36:33,720][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last 10 exchange futures (total: 2): [17:36:33,720][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000, 1db622e7-f722-4b86-a09e-d046cd900002], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967393534, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967393534, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=2, nodeId8=801bd00d, msg=null, type=NODE_JOINED, tstamp=1490967373450], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], nodeId=801bd00d, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373450, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967393534, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=2, res=AffinityTopologyVersion [topVer=2, minorTopVer=0], startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE]] [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transactions: [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending explicit locks: [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending cache futures: [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending atomic cache futures: [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending data streamer futures: [17:36:33,721][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transaction deadlock detection futures: [17:36:33,722][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, idx=0, connCnt=-1], msgsSent=7, msgsAckedByRmt=0, msgsRcvd=7, lastAcked=0, reserveCnt=1, descIdHash=971956974] Communication SPI clients: [node=1eb76302-be6c-4ad7-bf29-a6476d100000, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@4e4a0731, idx=0, bytesRcvd=23920, bytesSent=5068, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=1043732450, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51076, createTime=1490967373450, closeTime=0, bytesSent=5068, bytesRcvd=23920, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967373450, lastSndTime=1490967375171, lastRcvTime=1490967376680, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967373450, closed=false, connIdx=0]]] [17:36:33,722][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] NIO server statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0] [17:36:33,723][WARN ][grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=1, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:36:33,723][WARN ][grid-nio-worker-tcp-comm-3-#846%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:36:33,723][WARN ][grid-nio-worker-tcp-comm-2-#845%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:36:33,725][WARN ][grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=0, keysCnt=1, bytesRcvd=23920, bytesRcvd0=0, bytesSent=5068, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51076, locAddr=/127.0.0.1:45011, msgsSent=7, msgsAckedByRmt=0, descIdHash=971956974, msgsRcvd=7, lastAcked=0, descIdHash=971956974, bytesRcvd=23920, bytesRcvd0=0, bytesSent=5068, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null]], lastRead=true]] [17:36:36,213][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=2479] [17:36:36,215][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest$TestFailedNodesSpi] Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=1] [17:36:36,221][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-sys-cache, mode=REPLICATED] [17:36:36,221][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-atomics-sys-cache, mode=PARTITIONED] [17:36:36,223][INFO ][exchange-worker-#892%tcp.TcpDiscoverySelfTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, node=1db622e7-f722-4b86-a09e-d046cd900002] [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Performance suggestions for grid 'tcp.TcpDiscoverySelfTest2' (fix if possible) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Disable checkpoints (remove 'checkpointSpi' from configuration) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Disable grid events (remove 'includeEventTypes' from configuration) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Use default binary marshaller (do not set 'marshaller' explicitly) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] ^-- Disable assertions (remove '-ea' from JVM options) [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [17:36:36,226][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] [17:36:36,227][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest2] >>> +------------------------------------------+ >>> Ignite ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> +------------------------------------------+ >>> OS name: Mac OS X 10.12.4 x86_64 >>> CPU(s): 8 >>> Heap: 3.0GB >>> VM name: 7858@mbp.local >>> Ignite instance name: tcp.TcpDiscoverySelfTest2 >>> Local node [ID=1DB622E7-F722-4B86-A09E-D046CD900002, order=1, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:45012 TCP:47502 [17:36:36,227][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridDiscoveryManager] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, heap=3.0GB] [17:36:36,227][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][root] Node started with the following configuration [id=1db622e7-f722-4b86-a09e-d046cd900002, marshaller=org.apache.ignite.marshaller.optimized.OptimizedMarshaller@202ddbe4, binaryCfg=null, lateAff=true] [17:36:36,230][WARN ][stop-node-1][TcpDiscoverySelfTest$TestFailedNodesSpi] Runnable tasks outlived thread pool executor service [owner=ServerImpl, tasks=[o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$8@2f1f9005]] [17:36:36,586][WARN ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:36,586][INFO ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Topology snapshot [ver=4, servers=2, clients=0, CPUs=8, heap=3.0GB] [17:36:46,225][INFO ][stop-node-1][GridCacheProcessor] Stopped cache: ignite-sys-cache [17:36:46,225][INFO ][stop-node-1][GridCacheProcessor] Stopped cache: ignite-atomics-sys-cache [17:36:46,227][INFO ][stop-node-1][TcpDiscoverySelfTest2] >>> +-----------------------------------------------------+ >>> Ignite ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV stopped OK >>> +-----------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoverySelfTest2 >>> Grid uptime: 00:00:10:005 [17:36:46,224][WARN ][disco-pool-#888%tcp.TcpDiscoverySelfTest2%][TcpDiscoverySelfTest$TestFailedNodesSpi] Timed out waiting for message to be read (most probably, the reason is in long GC pauses on remote node) [curTimeout=2484] [17:36:46,351][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][root] Try start new node. [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.0.0-SNAPSHOT#19700101-sha1:DEV >>> 2017 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Config URL: n/a [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Daemon mode: off [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] OS: Mac OS X 10.12.4 x86_64 [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] OS user: yzhdanov [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] PID: 7858 [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Language runtime: Java Platform API Specification ver. 1.8 [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] VM information: Java(TM) SE Runtime Environment 1.8.0-b132 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.0-b70 [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] VM total memory: 2.9GB [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Remote Management [restart: off, REST: off, JMX (remote: off)] [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] IGNITE_HOME=/Users/yzhdanov/projects/incubator-ignite [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] VM arguments: [-ea, -Xmx3g, -Xms3g, -Didea.launcher.port=7533, -Didea.launcher.bin.path=/Applications/IntelliJ IDEA CE.app/Contents/bin, -Dfile.encoding=UTF-8] [17:36:46,355][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Configured caches ['ignite-sys-cache', 'ignite-atomics-sys-cache'] [17:36:46,355][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [17:36:46,356][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] 3-rd party licenses can be found at: /Users/yzhdanov/projects/incubator-ignite/libs/licenses [17:36:46,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] Configured plugins: [17:36:46,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- TestPlatformPlugin 1.0 [17:36:46,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] ^-- - [17:36:46,358][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][IgnitePluginProcessor] [17:36:46,360][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [17:36:46,359][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45012, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [17:36:46,365][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Security status [authentication=off, tls/ssl=off] [17:36:46,364][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [17:36:46,365][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][NoopSwapSpaceSpi] Swap space is disabled. To enable use FileSwapSpaceSpi. [17:36:46,370][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Non-loopback local IPs: 172.25.4.215, fe80:0:0:0:394f:9f9f:4d7e:efc%utun0, fe80:0:0:0:c53:b736:8469:7030%en1 [17:36:46,370][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySelfTest3] Enabled local MACs: 1040F39C4BE4 [17:36:46,372][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySpi] Successfully bound to TCP port [port=47502, localHost=/127.0.0.1, locNodeId=ebc55309-1ccf-4270-a60b-3e3f79600003] [17:36:46,371][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][TcpDiscoverySpi] Network timeout is too low (at least 3000 ms recommended): 2500 [17:36:46,381][INFO ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:46,381][INFO ][disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%][GridDiscoveryManager] Topology snapshot [ver=5, servers=3, clients=0, CPUs=8, heap=3.0GB] [17:36:46,384][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-sys-cache, mode=REPLICATED] [17:36:46,385][INFO ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCacheProcessor] Started cache [name=ignite-atomics-sys-cache, mode=PARTITIONED] [17:36:46,386][INFO ][grid-nio-worker-tcp-comm-2-#818%tcp.TcpDiscoverySelfTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45010, rmtAddr=/127.0.0.1:51114] [17:36:46,386][WARN ][exchange-worker-#923%tcp.TcpDiscoverySelfTest3%][GridDhtAssignmentFetchFuture] Failed to request affinity assignment from remote node (node left grid, will continue to another node): TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:46,387][INFO ][grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51116] [17:36:46,389][WARN ][exchange-worker-#923%tcp.TcpDiscoverySelfTest3%][GridDhtAssignmentFetchFuture] Failed to request affinity assignment from remote node (node left grid, will continue to another node): TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false] [17:36:46,388][INFO ][grid-nio-worker-tcp-comm-0-#915%tcp.TcpDiscoverySelfTest3%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:51116, rmtAddr=/127.0.0.1:45011] [17:36:46,389][INFO ][grid-nio-worker-tcp-comm-3-#819%tcp.TcpDiscoverySelfTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45010, rmtAddr=/127.0.0.1:51118] [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=801bd00d-7408-4dc2-ac92-928b6a400001]. Dumping pending objects that might be the cause: [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Ready affinity version: AffinityTopologyVersion [topVer=2, minorTopVer=0] [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967413716, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending exchange futures: [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,726][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last 10 exchange futures (total: 4): [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967413727, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967413727, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=2, nodeId8=801bd00d, msg=null, type=NODE_JOINED, tstamp=1490967373450], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], nodeId=801bd00d, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373450, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967413727, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=2, res=AffinityTopologyVersion [topVer=2, minorTopVer=0], startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE]] [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transactions: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending explicit locks: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending cache futures: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending atomic cache futures: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending data streamer futures: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transaction deadlock detection futures: [17:36:53,727][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:36:53,733][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:36:53,733][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-atomics-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:36:53,733][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:36:53,733][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, idx=0, connCnt=-1], msgsSent=7, msgsAckedByRmt=7, msgsRcvd=7, lastAcked=7, reserveCnt=1, descIdHash=971956974] [key=ConnectionKey [nodeId=ebc55309-1ccf-4270-a60b-3e3f79600003, idx=0, connCnt=0], msgsSent=0, msgsAckedByRmt=0, msgsRcvd=2, lastAcked=0, reserveCnt=1, descIdHash=174555345] Communication SPI clients: [node=ebc55309-1ccf-4270-a60b-3e3f79600003, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@1a8df35a, idx=1, bytesRcvd=88, bytesSent=28, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=81796984, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=0, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51116, createTime=1490967406382, closeTime=0, bytesSent=28, bytesRcvd=88, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967406382, lastSndTime=1490967406382, lastRcvTime=1490967406382, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967406382, closed=false, connIdx=0]]] [node=1eb76302-be6c-4ad7-bf29-a6476d100000, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@4e4a0731, idx=0, bytesRcvd=23930, bytesSent=5078, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=1043732450, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51076, createTime=1490967373450, closeTime=0, bytesSent=5078, bytesRcvd=23930, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967373450, lastSndTime=1490967403370, lastRcvTime=1490967403315, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967373450, closed=false, connIdx=0]]] [17:36:53,733][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] NIO server statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0] [17:36:53,733][WARN ][grid-nio-worker-tcp-comm-2-#845%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:36:53,733][WARN ][grid-nio-worker-tcp-comm-3-#846%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:36:53,734][WARN ][grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=1, keysCnt=1, bytesRcvd=88, bytesRcvd0=0, bytesSent=28, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51116, locAddr=/127.0.0.1:45011, msgsSent=0, msgsAckedByRmt=0, descIdHash=174555345, msgsRcvd=2, lastAcked=0, descIdHash=174555345, bytesRcvd=88, bytesRcvd0=0, bytesSent=28, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=null, baseOff=0, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=false], state=0, hdrWritten=false], null, null, null, null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null, null]], lastRead=true]] [17:36:53,734][WARN ][grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=0, keysCnt=1, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51076, locAddr=/127.0.0.1:45011, msgsSent=7, msgsAckedByRmt=7, descIdHash=971956974, msgsRcvd=7, lastAcked=7, descIdHash=971956974, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null]], lastRead=true]] [17:37:13,737][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=801bd00d-7408-4dc2-ac92-928b6a400001]. Dumping pending objects that might be the cause: [17:37:13,737][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Ready affinity version: AffinityTopologyVersion [topVer=2, minorTopVer=0] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967433728, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending exchange futures: [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last 10 exchange futures (total: 4): [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,738][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967433738, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967433738, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=2, nodeId8=801bd00d, msg=null, type=NODE_JOINED, tstamp=1490967373450], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], nodeId=801bd00d, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373450, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967433738, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=2, res=AffinityTopologyVersion [topVer=2, minorTopVer=0], startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE]] [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transactions: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending explicit locks: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending cache futures: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending atomic cache futures: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending data streamer futures: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transaction deadlock detection futures: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-atomics-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:37:13,739][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, idx=0, connCnt=-1], msgsSent=7, msgsAckedByRmt=7, msgsRcvd=7, lastAcked=7, reserveCnt=1, descIdHash=971956974] [key=ConnectionKey [nodeId=ebc55309-1ccf-4270-a60b-3e3f79600003, idx=0, connCnt=0], msgsSent=0, msgsAckedByRmt=0, msgsRcvd=2, lastAcked=2, reserveCnt=1, descIdHash=174555345] Communication SPI clients: [node=ebc55309-1ccf-4270-a60b-3e3f79600003, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@1a8df35a, idx=1, bytesRcvd=88, bytesSent=38, bytesRcvd0=0, bytesSent0=10, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=81796984, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51116, createTime=1490967406382, closeTime=0, bytesSent=38, bytesRcvd=88, bytesSent0=10, bytesRcvd0=0, sndSchedTime=1490967406382, lastSndTime=1490967433367, lastRcvTime=1490967406382, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967406382, closed=false, connIdx=0]]] [node=1eb76302-be6c-4ad7-bf29-a6476d100000, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@4e4a0731, idx=0, bytesRcvd=23930, bytesSent=5078, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=1043732450, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51076, createTime=1490967373450, closeTime=0, bytesSent=5078, bytesRcvd=23930, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967373450, lastSndTime=1490967403370, lastRcvTime=1490967403315, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967373450, closed=false, connIdx=0]]] [17:37:13,740][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] NIO server statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0] [17:37:13,740][WARN ][grid-nio-worker-tcp-comm-3-#846%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:37:13,740][WARN ][grid-nio-worker-tcp-comm-2-#845%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:37:13,740][WARN ][grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=1, keysCnt=1, bytesRcvd=88, bytesRcvd0=0, bytesSent=38, bytesSent0=10] Connection info [in=true, rmtAddr=/127.0.0.1:51116, locAddr=/127.0.0.1:45011, msgsSent=0, msgsAckedByRmt=0, descIdHash=174555345, msgsRcvd=2, lastAcked=2, descIdHash=174555345, bytesRcvd=88, bytesRcvd0=0, bytesSent=38, bytesSent0=10, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=null, baseOff=0, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=false], state=0, hdrWritten=false], null, null, null, null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null, null]], lastRead=true]] [17:37:13,740][WARN ][grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=0, keysCnt=1, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51076, locAddr=/127.0.0.1:45011, msgsSent=7, msgsAckedByRmt=7, descIdHash=971956974, msgsRcvd=7, lastAcked=7, descIdHash=971956974, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null]], lastRead=true]] [17:37:26,387][WARN ][test-runner-#799%tcp.TcpDiscoverySelfTest%][GridCachePartitionExchangeManager] Failed to wait for initial partition map exchange. Possible reasons are: ^-- Transactions in deadlock. ^-- Long running transactions (ignore if this is the case). ^-- Unreleased explicit locks. [17:37:33,743][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], node=801bd00d-7408-4dc2-ac92-928b6a400001]. Dumping pending objects that might be the cause: [17:37:33,743][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Ready affinity version: AffinityTopologyVersion [topVer=2, minorTopVer=0] [17:37:33,743][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last exchange future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967453740, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,743][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending exchange futures: [17:37:33,744][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,744][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,744][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Last 10 exchange futures (total: 4): [17:37:33,744][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=5, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967406371], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], nodeId=ebc55309, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967406371, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,744][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=4, nodeId8=801bd00d, msg=Node failed: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_FAILED, tstamp=1490967396582], crd=null, exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nodeId=1db622e7, evt=NODE_FAILED], added=true, initFut=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT], init=false, lastVer=null, partReleaseFut=null, affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=0, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[], srvNodes=null, super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967396582, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=3, nodeId8=801bd00d, msg=Node joined: TcpDiscoveryNode [id=1db622e7-f722-4b86-a09e-d046cd900002, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1490967373695, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], type=NODE_JOINED, tstamp=1490967373707], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=1db622e7, evt=NODE_JOINED], added=true, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373707, endTime=1490967373707, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373707, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967453740, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=0, res=null, startTime=1490967373707, endTime=0, ignoreInterrupts=false, state=INIT]] [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] >>> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967453740, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false], topVer=2, nodeId8=801bd00d, msg=null, type=NODE_JOINED, tstamp=1490967373450], crd=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], nodeId=801bd00d, evt=NODE_JOINED], added=false, initFut=GridFutureAdapter [resFlag=2, res=true, startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE], init=true, lastVer=null, partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null, futs=[true, true, true, true]], affChangeMsg=null, skipPreload=false, clientOnlyExchange=false, initTs=1490967373450, centralizedAff=false, forcedRebFut=null, evtLatch=0, remaining=[1eb76302-be6c-4ad7-bf29-a6476d100000], srvNodes=[TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], TcpDiscoveryNode [id=801bd00d-7408-4dc2-ac92-928b6a400001, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1490967453740, loc=true, ver=2.0.0#19700101-sha1:00000000, isClient=false]], super=GridFutureAdapter [resFlag=2, res=AffinityTopologyVersion [topVer=2, minorTopVer=0], startTime=1490967373450, endTime=1490967373661, ignoreInterrupts=false, state=DONE]] [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transactions: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending explicit locks: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending cache futures: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending atomic cache futures: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending data streamer futures: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridCachePartitionExchangeManager] Pending transaction deadlock detection futures: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] Pending affinity ready futures [cache=ignite-atomics-sys-cache, lastVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]]: [17:37:33,745][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][GridAffinityAssignmentCache] >>> AffinityReadyFuture [reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [17:37:33,746][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] Communication SPI recovery descriptors: [key=ConnectionKey [nodeId=1eb76302-be6c-4ad7-bf29-a6476d100000, idx=0, connCnt=-1], msgsSent=7, msgsAckedByRmt=7, msgsRcvd=7, lastAcked=7, reserveCnt=1, descIdHash=971956974] [key=ConnectionKey [nodeId=ebc55309-1ccf-4270-a60b-3e3f79600003, idx=0, connCnt=0], msgsSent=0, msgsAckedByRmt=0, msgsRcvd=2, lastAcked=2, reserveCnt=1, descIdHash=174555345] Communication SPI clients: [node=ebc55309-1ccf-4270-a60b-3e3f79600003, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@1a8df35a, idx=1, bytesRcvd=88, bytesSent=38, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=81796984, interrupted=false, runner=grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=0, resendCnt=0, rcvCnt=2, sentCnt=0, reserved=true, lastAck=2, nodeLeft=false, node=TcpDiscoveryNode [id=ebc55309-1ccf-4270-a60b-3e3f79600003, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47502], discPort=47502, order=5, intOrder=4, lastExchangeTime=1490967406371, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=0, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51116, createTime=1490967406382, closeTime=0, bytesSent=38, bytesRcvd=88, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967437742, lastSndTime=1490967433367, lastRcvTime=1490967437742, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967406382, closed=false, connIdx=0]]] [node=1eb76302-be6c-4ad7-bf29-a6476d100000, client=GridTcpNioCommunicationClient [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [selector=sun.nio.ch.KQueueSelectorImpl@4e4a0731, idx=0, bytesRcvd=23930, bytesSent=5078, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=tcp.TcpDiscoverySelfTest1, finished=false, isCancelled=false, hashCode=1043732450, interrupted=false, runner=grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], outRecovery=GridNioRecoveryDescriptor [acked=7, resendCnt=0, rcvCnt=7, sentCnt=7, reserved=true, lastAck=7, nodeLeft=false, node=TcpDiscoveryNode [id=1eb76302-be6c-4ad7-bf29-a6476d100000, addrs=[127.0.0.1], sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1490967373450, loc=false, ver=2.0.0#19700101-sha1:00000000, isClient=false], connected=true, connectCnt=1, queueLimit=4096, reserveCnt=1, pairedConnections=false], super=GridNioSessionImpl [locAddr=/127.0.0.1:45011, rmtAddr=/127.0.0.1:51076, createTime=1490967373450, closeTime=0, bytesSent=5078, bytesRcvd=23930, bytesSent0=0, bytesRcvd0=0, sndSchedTime=1490967433738, lastSndTime=1490967403370, lastRcvTime=1490967433738, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@7001729e, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]], super=GridAbstractCommunicationClient [lastUsed=1490967373450, closed=false, connIdx=0]]] [17:37:33,746][WARN ][exchange-worker-#851%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] NIO server statistics [readerSesBalanceCnt=0, writerSesBalanceCnt=0] [17:37:33,746][WARN ][grid-nio-worker-tcp-comm-2-#845%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=2, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:37:33,746][WARN ][grid-nio-worker-tcp-comm-3-#846%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=3, keysCnt=0, bytesRcvd=0, bytesRcvd0=0, bytesSent=0, bytesSent0=0] [17:37:33,747][WARN ][grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=1, keysCnt=1, bytesRcvd=88, bytesRcvd0=0, bytesSent=38, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51116, locAddr=/127.0.0.1:45011, msgsSent=0, msgsAckedByRmt=0, descIdHash=174555345, msgsRcvd=2, lastAcked=2, descIdHash=174555345, bytesRcvd=88, bytesRcvd0=0, bytesSent=38, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=null, baseOff=0, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=false], state=0, hdrWritten=false], null, null, null, null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314477733888, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null, null]], lastRead=true]] [17:37:33,747][WARN ][grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%][TcpCommunicationSpi] >> Selector info [idx=0, keysCnt=1, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0] Connection info [in=true, rmtAddr=/127.0.0.1:51076, locAddr=/127.0.0.1:45011, msgsSent=7, msgsAckedByRmt=7, descIdHash=971956974, msgsRcvd=7, lastAcked=7, descIdHash=971956974, bytesRcvd=23930, bytesRcvd0=0, bytesSent=5078, bytesSent0=0, opQueueSize=0, msgWriter=DirectMessageWriter [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314451119104, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0, hdrWritten=false], null, null, null, null, null, null]]], msgReader=DirectMessageReader [state=DirectMessageState [pos=0, stack=[StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], StateItem [stream=DirectByteBufferStreamImplV2 [buf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], baseOff=140314453136896, arrOff=-1, tmpArrOff=0, tmpArrBytes=0, msgTypeDone=false, msg=null, mapIt=null, it=null, arrPos=-1, keyDone=false, readSize=-1, readItems=0, prim=0, primShift=0, uuidState=0, uuidMost=0, uuidLeast=0, uuidLocId=0, lastFinished=true], state=0], null, null, null, null, null, null]], lastRead=true]] 2017-03-31 16:37:39 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode): "sys-#932%tcp.TcpDiscoverySelfTest1%" #2145 prio=5 os_prio=31 tid=0x00007f9d859c6800 nid=0x16913 waiting on condition [0x0000700004b56000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#931%tcp.TcpDiscoverySelfTest0%" #2144 prio=5 os_prio=31 tid=0x00007f9d80c60800 nid=0x16c13 waiting on condition [0x0000700004a53000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#930%tcp.TcpDiscoverySelfTest1%" #2143 prio=5 os_prio=31 tid=0x00007f9d85a3c000 nid=0x18313 waiting on condition [0x0000700004950000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#929%tcp.TcpDiscoverySelfTest0%" #2142 prio=5 os_prio=31 tid=0x00007f9d85a38800 nid=0x11723 waiting on condition [0x000070000484d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#928%tcp.TcpDiscoverySelfTest1%" #2141 prio=5 os_prio=31 tid=0x00007f9d85a38000 nid=0xeb27 waiting on condition [0x000070000474a000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#927%tcp.TcpDiscoverySelfTest0%" #2140 prio=5 os_prio=31 tid=0x00007f9d859be000 nid=0x11623 waiting on condition [0x0000700004647000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#926%tcp.TcpDiscoverySelfTest0%" #2139 prio=5 os_prio=31 tid=0x00007f9d88970000 nid=0x610f waiting on condition [0x000070000b082000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#925%tcp.TcpDiscoverySelfTest1%" #2138 prio=5 os_prio=31 tid=0x00007f9d85934000 nid=0x12c1b waiting on condition [0x000070000af7f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#924%tcp.TcpDiscoverySelfTest0%" #2137 prio=5 os_prio=31 tid=0x00007f9d80dd2800 nid=0xff1f waiting on condition [0x000070000ae7c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "exchange-worker-#923%tcp.TcpDiscoverySelfTest3%" #2136 prio=5 os_prio=31 tid=0x00007f9d80c5b800 nid=0x16613 waiting on condition [0x000070000ad79000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705caa4c0> (a org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtAssignmentFetchFuture) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:161) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:119) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinity(CacheAffinitySharedManager.java:984) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.fetchAffinityOnJoin(CacheAffinitySharedManager.java:966) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onServerJoin(CacheAffinitySharedManager.java:885) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onServerNodeEvent(GridDhtPartitionsExchangeFuture.java:685) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:499) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1713) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-time-server-reader-#922%tcp.TcpDiscoverySelfTest3%" #2135 prio=10 os_prio=31 tid=0x00007f9d88ad5000 nid=0x19b0f runnable [0x000070000ac76000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <0x0000000705ca46d0> (a java.net.PlainDatagramSocketImpl) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143) - locked <0x0000000705ca46d0> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:781) - locked <0x0000000705ca46a8> (a java.net.DatagramPacket) - locked <0x0000000705ca46f8> (a java.net.DatagramSocket) at org.apache.ignite.internal.processors.clock.GridClockServer$ReadWorker.body(GridClockServer.java:200) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "disco-event-worker-#921%tcp.TcpDiscoverySelfTest3%" #2134 prio=5 os_prio=31 tid=0x00007f9d87816800 nid=0x17217 waiting on condition [0x000070000ab73000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705ca6648> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2256) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2238) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "tcp-disco-ip-finder-cleaner-#1168%tcp.TcpDiscoverySelfTest3%" #2133 prio=10 os_prio=31 tid=0x00007f9d87816000 nid=0x1b90f waiting on condition [0x000070000aa70000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1784) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-sock-reader-#1167%tcp.TcpDiscoverySelfTest1%" #2132 prio=10 os_prio=31 tid=0x00007f9d87815000 nid=0x1821b runnable [0x000070000a96d000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x0000000705cae658> (a java.io.BufferedInputStream) at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.(ObjectInputStream.java:299) at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:39) at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:117) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9688) at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5722) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-sock-reader-#1166%tcp.TcpDiscoverySelfTest3%" #2131 prio=10 os_prio=31 tid=0x00007f9d88ac8000 nid=0x1c113 runnable [0x000070000a86a000] java.lang.Thread.State: RUNNABLE at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:132) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1565) at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5919) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-srvr-#1164%tcp.TcpDiscoverySelfTest3%" #2129 prio=10 os_prio=31 tid=0x00007f9d80c59800 nid=0x1671f runnable [0x000070000a052000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5426) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-msg-worker-#1163%tcp.TcpDiscoverySelfTest3%" #2128 prio=10 os_prio=31 tid=0x00007f9d87814800 nid=0x17a3f runnable [0x0000700009e4c000] java.lang.Thread.State: RUNNABLE at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705e42a50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6392) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2435) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-data-loader-flusher-#920%tcp.TcpDiscoverySelfTest3%" #2127 prio=5 os_prio=31 tid=0x00007f9d88800000 nid=0x16417 waiting on condition [0x0000700009d49000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705ca4b48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:107) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-sharedfs-timeout-worker-#1162%tcp.TcpDiscoverySelfTest3%" #2125 prio=5 os_prio=31 tid=0x00007f9d859bb800 nid=0x1213 in Object.wait() [0x0000700009a40000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) - locked <0x0000000705ca68d0> (a java.lang.Object) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-comm-worker-#1161%tcp.TcpDiscoverySelfTest3%" #2124 prio=5 os_prio=31 tid=0x00007f9d80c63800 nid=0x930f waiting on condition [0x000070000993d000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705caacc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3833) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-nio-worker-tcp-comm-3-#918%tcp.TcpDiscoverySelfTest3%" #2123 prio=5 os_prio=31 tid=0x00007f9d80c62800 nid=0x1320f runnable [0x000070000983a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000705ca6ab0> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x0000000705ca6aa0> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000705ca6ad0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-2-#917%tcp.TcpDiscoverySelfTest3%" #2122 prio=5 os_prio=31 tid=0x00007f9d859bc800 nid=0x1310f runnable [0x0000700009737000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000705ca5798> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x0000000705ca5788> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000705ca57b8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-1-#916%tcp.TcpDiscoverySelfTest3%" #2121 prio=5 os_prio=31 tid=0x00007f9d80ce6000 nid=0x1fd0f runnable [0x0000700009634000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000705ca6cd0> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x0000000705ca6cc0> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000705ca5fb8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-0-#915%tcp.TcpDiscoverySelfTest3%" #2120 prio=5 os_prio=31 tid=0x00007f9d80ce5800 nid=0x2090f runnable [0x0000700009531000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000705d7fb70> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x0000000705ca7d68> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000705d7fac0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "nio-acceptor-#914%tcp.TcpDiscoverySelfTest3%" #2119 prio=5 os_prio=31 tid=0x00007f9d80ce4800 nid=0x810f runnable [0x000070000942e000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000705cb1a70> (a sun.nio.ch.Util$2) - locked <0x0000000705cb1a60> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000705cb18c0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2445) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2417) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-timeout-worker-#913%tcp.TcpDiscoverySelfTest3%" #2118 prio=5 os_prio=31 tid=0x00007f9d80ce4000 nid=0x720f in Object.wait() [0x000070000932b000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:184) - locked <0x0000000705ca7db8> (a java.lang.Object) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "pub-#912%tcp.TcpDiscoverySelfTest3%" #2117 prio=5 os_prio=31 tid=0x00007f9d80c7e800 nid=0x830f waiting on condition [0x0000700009228000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "pub-#911%tcp.TcpDiscoverySelfTest3%" #2116 prio=5 os_prio=31 tid=0x00007f9d859b9800 nid=0x780f waiting on condition [0x0000700009125000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "pub-#910%tcp.TcpDiscoverySelfTest3%" #2115 prio=5 os_prio=31 tid=0x00007f9d87819000 nid=0x770f waiting on condition [0x0000700009022000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "pub-#909%tcp.TcpDiscoverySelfTest3%" #2114 prio=5 os_prio=31 tid=0x00007f9d87818800 nid=0x7a0f waiting on condition [0x0000700008f1f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "pub-#908%tcp.TcpDiscoverySelfTest3%" #2113 prio=5 os_prio=31 tid=0x00007f9d87817800 nid=0x840f waiting on condition [0x0000700008e1c000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "pub-#907%tcp.TcpDiscoverySelfTest3%" #2112 prio=5 os_prio=31 tid=0x00007f9d859ba800 nid=0xb1ab waiting on condition [0x0000700008d19000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705d7f288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-stripe-7-#906%tcp.TcpDiscoverySelfTest3%" #2111 prio=5 os_prio=31 tid=0x00007f9d859b9000 nid=0x10d1f waiting on condition [0x0000700008c16000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-6-#905%tcp.TcpDiscoverySelfTest3%" #2110 prio=5 os_prio=31 tid=0x00007f9d859b8000 nid=0x18f1f waiting on condition [0x0000700008b13000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-5-#904%tcp.TcpDiscoverySelfTest3%" #2109 prio=5 os_prio=31 tid=0x00007f9d859c5800 nid=0x21013 waiting on condition [0x0000700008a10000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-4-#903%tcp.TcpDiscoverySelfTest3%" #2108 prio=5 os_prio=31 tid=0x00007f9d859c5000 nid=0xf923 waiting on condition [0x000070000890d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-3-#902%tcp.TcpDiscoverySelfTest3%" #2107 prio=5 os_prio=31 tid=0x00007f9d859c4000 nid=0x1121f waiting on condition [0x000070000880a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-2-#901%tcp.TcpDiscoverySelfTest3%" #2106 prio=5 os_prio=31 tid=0x00007f9d80aab000 nid=0xe92b waiting on condition [0x0000700008707000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-1-#900%tcp.TcpDiscoverySelfTest3%" #2105 prio=5 os_prio=31 tid=0x00007f9d80978000 nid=0x15927 waiting on condition [0x0000700008604000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-0-#899%tcp.TcpDiscoverySelfTest3%" #2104 prio=5 os_prio=31 tid=0x00007f9d859c3800 nid=0xc12b waiting on condition [0x0000700008501000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-#898%tcp.TcpDiscoverySelfTest1%" #2102 prio=5 os_prio=31 tid=0x00007f9d88ac8800 nid=0x16813 waiting on condition [0x000070000a767000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#897%tcp.TcpDiscoverySelfTest0%" #2101 prio=5 os_prio=31 tid=0x00007f9d859b6800 nid=0x8f0f waiting on condition [0x000070000a664000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "srvc-deploy-#859%tcp.TcpDiscoverySelfTest1%" #2037 prio=5 os_prio=31 tid=0x00007f9d8180e000 nid=0x13027 waiting on condition [0x00007000083fe000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033e3390> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "aff-#854%tcp.TcpDiscoverySelfTest1%" #2032 prio=5 os_prio=31 tid=0x00007f9d831dc000 nid=0xc713 waiting on condition [0x0000700007eef000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033e3cd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "sys-#852%tcp.TcpDiscoverySelfTest1%" #2030 prio=5 os_prio=31 tid=0x00007f9d841b2800 nid=0x16a77 waiting on condition [0x0000700007ce9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033b1600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "exchange-worker-#851%tcp.TcpDiscoverySelfTest1%" #2029 prio=5 os_prio=31 tid=0x00007f9d841b1800 nid=0x16b17 waiting on condition [0x0000700007be6000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007055fa950> (a org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:189) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1719) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-time-server-reader-#850%tcp.TcpDiscoverySelfTest1%" #2028 prio=10 os_prio=31 tid=0x00007f9d81339800 nid=0x16117 runnable [0x0000700007ae3000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <0x00000007033e34a0> (a java.net.PlainDatagramSocketImpl) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143) - locked <0x00000007033e34a0> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:781) - locked <0x0000000705845458> (a java.net.DatagramPacket) - locked <0x0000000703596ac0> (a java.net.DatagramSocket) at org.apache.ignite.internal.processors.clock.GridClockServer$ReadWorker.body(GridClockServer.java:200) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "disco-event-worker-#849%tcp.TcpDiscoverySelfTest1%" #2027 prio=5 os_prio=31 tid=0x00007f9d81a0b800 nid=0x21f23 waiting on condition [0x00007000079e0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000703596c98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2256) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2238) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "tcp-disco-ip-finder-cleaner-#1136%tcp.TcpDiscoverySelfTest1%" #2026 prio=10 os_prio=31 tid=0x00007f9d82a71000 nid=0x13723 waiting on condition [0x00007000078dd000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1784) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-sock-reader-#1134%tcp.TcpDiscoverySelfTest1%" #2024 prio=10 os_prio=31 tid=0x00007f9d82ad6000 nid=0x1631b runnable [0x00007000076d6000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000007035714a0> (a java.io.BufferedInputStream) at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802) at java.io.ObjectInputStream.(ObjectInputStream.java:299) at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:39) at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:117) at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9688) at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5722) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-srvr-#1132%tcp.TcpDiscoverySelfTest1%" #2022 prio=10 os_prio=31 tid=0x00007f9d853e2800 nid=0x1a123 runnable [0x00007000075d4000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5426) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-msg-worker-#1131%tcp.TcpDiscoverySelfTest1%" #2021 prio=10 os_prio=31 tid=0x00007f9d80b1e800 nid=0x17017 runnable [0x00007000074d1000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.net.SocketInputStream.read(SocketInputStream.java:203) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.readReceipt(TcpDiscoverySpi.java:1645) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.sendMessageAcrossRing(ServerImpl.java:2963) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processStatusCheckMessage(ServerImpl.java:4718) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2550) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2349) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6397) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2435) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-data-loader-flusher-#848%tcp.TcpDiscoverySelfTest1%" #2020 prio=5 os_prio=31 tid=0x00007f9d829ac800 nid=0x8817 waiting on condition [0x00007000073ce000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007033dac20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:107) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-sharedfs-timeout-worker-#1130%tcp.TcpDiscoverySelfTest1%" #2018 prio=5 os_prio=31 tid=0x00007f9d83763800 nid=0x21d13 in Object.wait() [0x00007000072cb000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) - locked <0x0000000703571668> (a java.lang.Object) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-comm-worker-#1129%tcp.TcpDiscoverySelfTest1%" #2017 prio=5 os_prio=31 tid=0x00007f9d831e3000 nid=0xfc23 waiting on condition [0x00007000071c8000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000703699048> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3833) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-nio-worker-tcp-comm-3-#846%tcp.TcpDiscoverySelfTest1%" #2016 prio=5 os_prio=31 tid=0x00007f9d8490a800 nid=0x1512b runnable [0x00007000070c5000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070360bea8> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x000000070360cf18> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070360bdf8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-2-#845%tcp.TcpDiscoverySelfTest1%" #2015 prio=5 os_prio=31 tid=0x00007f9d84acd000 nid=0xbe13 runnable [0x0000700006fc2000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070360d400> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x000000070360d460> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070360d350> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-1-#844%tcp.TcpDiscoverySelfTest1%" #2014 prio=5 os_prio=31 tid=0x00007f9d840b9000 nid=0x1851f runnable [0x0000700006ebf000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070360d948> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x000000070360d9a8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070360d898> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-0-#843%tcp.TcpDiscoverySelfTest1%" #2013 prio=5 os_prio=31 tid=0x00007f9d842b9000 nid=0x13823 runnable [0x0000700006dbc000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000703697a00> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x0000000703698bd0> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000703697950> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "nio-acceptor-#842%tcp.TcpDiscoverySelfTest1%" #2012 prio=5 os_prio=31 tid=0x00007f9d81669800 nid=0xbf1b runnable [0x0000700006cb9000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000007033e37f0> (a sun.nio.ch.Util$2) - locked <0x00000007033e37e0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007033e3640> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2445) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2417) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-timeout-worker-#841%tcp.TcpDiscoverySelfTest1%" #2011 prio=5 os_prio=31 tid=0x00007f9d831e2800 nid=0x15b17 in Object.wait() [0x0000700006bb6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:184) - locked <0x000000070351df40> (a java.lang.Object) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "sys-stripe-7-#834%tcp.TcpDiscoverySelfTest1%" #2004 prio=5 os_prio=31 tid=0x00007f9d831e1800 nid=0x18e23 waiting on condition [0x00007000064a1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-6-#833%tcp.TcpDiscoverySelfTest1%" #2003 prio=5 os_prio=31 tid=0x00007f9d8315f800 nid=0x1731b waiting on condition [0x000070000639e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-5-#832%tcp.TcpDiscoverySelfTest1%" #2002 prio=5 os_prio=31 tid=0x00007f9d8315e800 nid=0xfa23 waiting on condition [0x000070000629b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-4-#831%tcp.TcpDiscoverySelfTest1%" #2001 prio=5 os_prio=31 tid=0x00007f9d8322c800 nid=0x1b30f waiting on condition [0x0000700006198000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-3-#830%tcp.TcpDiscoverySelfTest1%" #2000 prio=5 os_prio=31 tid=0x00007f9d8322c000 nid=0x1b50f waiting on condition [0x0000700006095000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-2-#829%tcp.TcpDiscoverySelfTest1%" #1999 prio=5 os_prio=31 tid=0x00007f9d83185800 nid=0xfe23 waiting on condition [0x0000700005f92000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-1-#828%tcp.TcpDiscoverySelfTest1%" #1998 prio=5 os_prio=31 tid=0x00007f9d85841800 nid=0x21123 waiting on condition [0x0000700005e8f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-0-#827%tcp.TcpDiscoverySelfTest1%" #1997 prio=5 os_prio=31 tid=0x00007f9d83057000 nid=0xc61b waiting on condition [0x0000700005d8c000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "srvc-deploy-#826%tcp.TcpDiscoverySelfTest0%" #1995 prio=5 os_prio=31 tid=0x00007f9d81052800 nid=0x1002b waiting on condition [0x0000700005c89000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070340ec08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:744) "exchange-worker-#825%tcp.TcpDiscoverySelfTest0%" #1994 prio=5 os_prio=31 tid=0x00007f9d809aa000 nid=0xce23 waiting on condition [0x0000700005b86000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070374ca30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1684) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-time-coordinator-#824%tcp.TcpDiscoverySelfTest0%" #1993 prio=10 os_prio=31 tid=0x00007f9d80c61800 nid=0x20f0f in Object.wait() [0x0000700005a83000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.internal.processors.clock.GridClockSyncProcessor$TimeCoordinator.body(GridClockSyncProcessor.java:404) - locked <0x000000070374e148> (a org.apache.ignite.internal.processors.clock.GridClockSyncProcessor$TimeCoordinator) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-time-server-reader-#823%tcp.TcpDiscoverySelfTest0%" #1992 prio=10 os_prio=31 tid=0x00007f9d8197e000 nid=0xc30f runnable [0x0000700005980000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <0x000000070340f0b0> (a java.net.PlainDatagramSocketImpl) at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143) - locked <0x000000070340f0b0> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:781) - locked <0x00000007038451f8> (a java.net.DatagramPacket) - locked <0x000000070374e228> (a java.net.DatagramSocket) at org.apache.ignite.internal.processors.clock.GridClockServer$ReadWorker.body(GridClockServer.java:200) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "disco-event-worker-#822%tcp.TcpDiscoverySelfTest0%" #1991 prio=5 os_prio=31 tid=0x00007f9d814f9000 nid=0xe113 waiting on condition [0x000070000587d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070344fcd8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2256) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2238) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "tcp-disco-ip-finder-cleaner-#1128%tcp.TcpDiscoverySelfTest0%" #1990 prio=10 os_prio=31 tid=0x00007f9d84008800 nid=0x16013 waiting on condition [0x000070000577a000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1784) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-srvr-#1127%tcp.TcpDiscoverySelfTest0%" #1989 prio=10 os_prio=31 tid=0x00007f9d85840800 nid=0x16213 runnable [0x0000700005677000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5426) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-disco-msg-worker-#1126%tcp.TcpDiscoverySelfTest0%" #1988 prio=10 os_prio=31 tid=0x00007f9d8189b000 nid=0x13f1f waiting on condition [0x0000700005574000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000703845d20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6392) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2435) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-data-loader-flusher-#821%tcp.TcpDiscoverySelfTest0%" #1987 prio=5 os_prio=31 tid=0x00007f9d83056800 nid=0x1411f waiting on condition [0x0000700005471000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x000000070377eb20> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:107) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-sharedfs-timeout-worker-#1125%tcp.TcpDiscoverySelfTest0%" #1985 prio=5 os_prio=31 tid=0x00007f9d8485c000 nid=0xb50f in Object.wait() [0x000070000536e000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) - locked <0x000000070384d700> (a java.lang.Object) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "tcp-comm-worker-#1124%tcp.TcpDiscoverySelfTest0%" #1984 prio=5 os_prio=31 tid=0x00007f9d81825800 nid=0xe013 waiting on condition [0x000070000526b000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000703898548> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:3833) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) "grid-nio-worker-tcp-comm-3-#819%tcp.TcpDiscoverySelfTest0%" #1983 prio=5 os_prio=31 tid=0x00007f9d8480b000 nid=0x19413 runnable [0x0000700005168000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070377f750> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x000000070377f7b0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070377f6a0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-2-#818%tcp.TcpDiscoverySelfTest0%" #1982 prio=5 os_prio=31 tid=0x00007f9d831de800 nid=0x13d1b runnable [0x0000700005065000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070377ff68> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x000000070377ffc8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070377feb8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-1-#817%tcp.TcpDiscoverySelfTest0%" #1981 prio=5 os_prio=31 tid=0x00007f9d81824800 nid=0x11f1b runnable [0x0000700004f62000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000703780498> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x00000007038be720> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007037803e8> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-nio-worker-tcp-comm-0-#816%tcp.TcpDiscoverySelfTest0%" #1980 prio=5 os_prio=31 tid=0x00007f9d84892800 nid=0x19727 runnable [0x0000700004e5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000007038be930> (a org.apache.ignite.internal.util.nio.SelectedSelectionKeySet) - locked <0x00000007038be920> (a java.util.Collections$UnmodifiableSet) - locked <0x00000007038be950> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:1863) at org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1573) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "nio-acceptor-#815%tcp.TcpDiscoverySelfTest0%" #1979 prio=5 os_prio=31 tid=0x00007f9d850e8000 nid=0x19a13 runnable [0x0000700004d5c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x000000070340f400> (a sun.nio.ch.Util$2) - locked <0x000000070340f3f0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000070340f250> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2445) at org.apache.ignite.internal.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2417) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "grid-timeout-worker-#814%tcp.TcpDiscoverySelfTest0%" #1978 prio=5 os_prio=31 tid=0x00007f9d837f7000 nid=0x19913 in Object.wait() [0x0000700004c59000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:184) - locked <0x0000000703896c00> (a java.lang.Object) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:744) "sys-stripe-7-#807%tcp.TcpDiscoverySelfTest0%" #1971 prio=5 os_prio=31 tid=0x00007f9d84892000 nid=0x1021f waiting on condition [0x0000700004544000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-6-#806%tcp.TcpDiscoverySelfTest0%" #1970 prio=5 os_prio=31 tid=0x00007f9d82ab2800 nid=0x6747 waiting on condition [0x0000700004441000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-5-#805%tcp.TcpDiscoverySelfTest0%" #1969 prio=5 os_prio=31 tid=0x00007f9d84102000 nid=0x21a0f waiting on condition [0x000070000433e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-4-#804%tcp.TcpDiscoverySelfTest0%" #1968 prio=5 os_prio=31 tid=0x00007f9d82aa8000 nid=0x2190f waiting on condition [0x000070000423b000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-3-#803%tcp.TcpDiscoverySelfTest0%" #1967 prio=5 os_prio=31 tid=0x00007f9d84015800 nid=0x6c13 waiting on condition [0x0000700004138000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-2-#802%tcp.TcpDiscoverySelfTest0%" #1966 prio=5 os_prio=31 tid=0x00007f9d82941800 nid=0x1401f waiting on condition [0x0000700004035000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-1-#801%tcp.TcpDiscoverySelfTest0%" #1965 prio=5 os_prio=31 tid=0x00007f9d82941000 nid=0x1151f waiting on condition [0x0000700003f32000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "sys-stripe-0-#800%tcp.TcpDiscoverySelfTest0%" #1964 prio=5 os_prio=31 tid=0x00007f9d80b09800 nid=0x8713 waiting on condition [0x0000700003e2f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:581) at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:477) at java.lang.Thread.run(Thread.java:744) "test-runner-#799%tcp.TcpDiscoverySelfTest%" #1963 prio=5 os_prio=31 tid=0x00007f9d83174800 nid=0x5b1f waiting on condition [0x0000700002766000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000705caa588> (a org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:189) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:130) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStart0(GridCachePartitionExchangeManager.java:445) at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStart(GridCacheSharedManagerAdapter.java:101) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:848) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:989) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1846) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1637) - locked <0x0000000705dab998> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1074) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:572) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:798) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:782) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:770) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:747) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySelfTest.testFailedNodes4(TcpDiscoverySelfTest.java:1797) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at junit.framework.TestCase.runTest(TestCase.java:176) at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:1899) at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:121) at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1814) at java.lang.Thread.run(Thread.java:744) "ignite-clock-for-tests" #11 daemon prio=10 os_prio=31 tid=0x00007f9d81098000 nid=0x5903 waiting on condition [0x0000700003d2c000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.ignite.internal.util.GridTestClockTimer.run(GridTestClockTimer.java:41) at java.lang.Thread.run(Thread.java:744) "Monitor Ctrl-Break" #10 daemon prio=5 os_prio=31 tid=0x00007f9d83835800 nid=0x5703 runnable [0x0000700003c29000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked <0x0000000793aba450> (a java.io.InputStreamReader) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) - locked <0x0000000793aba450> (a java.io.InputStreamReader) at java.io.BufferedReader.readLine(BufferedReader.java:389) at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:84) at java.lang.Thread.run(Thread.java:744) "Service Thread" #9 daemon prio=9 os_prio=31 tid=0x00007f9d83811800 nid=0x5303 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C1 CompilerThread3" #8 daemon prio=9 os_prio=31 tid=0x00007f9d80812000 nid=0x5103 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f9d80811800 nid=0x4f03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f9d84802000 nid=0x4d03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f9d8302d800 nid=0x4b03 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f9d84800000 nid=0x4903 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f9d81017800 nid=0x3903 in Object.wait() [0x0000700003411000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142) - locked <0x000000070078d488> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f9d81016800 nid=0x3703 in Object.wait() [0x000070000330e000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157) - locked <0x000000070078d690> (a java.lang.ref.Reference$Lock) "main" #1 prio=5 os_prio=31 tid=0x00007f9d81008000 nid=0x1c03 in Object.wait() [0x00007000028ef000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) - locked <0x0000000703567960> (a org.apache.ignite.thread.IgniteThread) at org.apache.ignite.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1826) at junit.framework.TestCase.runBare(TestCase.java:141) at junit.framework.TestResult$1.protect(TestResult.java:122) at junit.framework.TestResult.runProtected(TestResult.java:142) at junit.framework.TestResult.run(TestResult.java:125) at junit.framework.TestCase.run(TestCase.java:129) at junit.framework.TestSuite.runTest(TestSuite.java:255) at junit.framework.TestSuite.run(TestSuite.java:250) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:237) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147) "VM Thread" os_prio=31 tid=0x00007f9d8304f800 nid=0x3503 runnable "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f9d80800800 nid=0x2503 runnable "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f9d8080c000 nid=0x2703 runnable "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f9d8080c800 nid=0x2903 runnable "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f9d81011000 nid=0x2b03 runnable "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007f9d81011800 nid=0x2d03 runnable "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007f9d81012000 nid=0x2f03 runnable "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007f9d83003000 nid=0x3103 runnable "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007f9d82801000 nid=0x3303 runnable "VM Periodic Task Thread" os_prio=31 tid=0x00007f9d8103e800 nid=0x5503 waiting on condition JNI global references: 53 Heap PSYoungGen total 1027584K, used 332793K [0x0000000780000000, 0x00000007c0000000, 0x00000007c0000000) eden space 1005568K, 33% used [0x0000000780000000,0x00000007944be728,0x00000007bd600000) from space 22016K, 1% used [0x00000007bea80000,0x00000007beac0000,0x00000007c0000000) to space 20992K, 0% used [0x00000007bd600000,0x00000007bd600000,0x00000007bea80000) ParOldGen total 2097152K, used 98799K [0x0000000700000000, 0x0000000780000000, 0x0000000780000000) object space 2097152K, 4% used [0x0000000700000000,0x000000070607bd68,0x0000000780000000) Metaspace used 26161K, capacity 27534K, committed 27776K, reserved 1073152K class space used 2840K, capacity 3133K, committed 3200K, reserved 1048576K