[2018-11-14 09:49:43,070][INFO ][main][root] >>> Starting test: TcpDiscoveryMultiThreadedTest#testCustomEventNodeRestart <<< [2018-11-14 09:49:43,074][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,075][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Config URL: n/a [2018-11-14 09:49:43,075][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest0, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@8757312, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@12ef6e23[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@3aa3937b, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@57b5b320, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@7632d514, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,081][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Daemon mode: off [2018-11-14 09:49:43,081][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,081][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] OS user: teamcity [2018-11-14 09:49:43,082][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] PID: 18275 [2018-11-14 09:49:43,082][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,082][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,082][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] VM total memory: 1.8GB [2018-11-14 09:49:43,082][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,083][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,083][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,083][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,086][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,087][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,087][WARN ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,088][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,090][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,090][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,090][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,091][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgnitePluginProcessor] [2018-11-14 09:49:43,092][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,092][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45180, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,093][WARN ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,095][WARN ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,095][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,097][WARN ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgniteCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files [2018-11-14 09:49:43,097][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,098][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,101][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [2018-11-14 09:49:43,104][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,104][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,106][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,106][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=/127.0.0.1, locNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,109][INFO ][disco-notifier-worker-#98282%tcp.TcpDiscoveryMultiThreadedTest0%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983102, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,110][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=true, crd=true, evt=NODE_JOINED, evtNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, customEvt=null, allowMerge=false] [2018-11-14 09:49:43,111][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,111][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][msg] Components activation performed in 0 ms. [2018-11-14 09:49:43,112][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,112][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCacheProcessor] Starting caches on local join performed in 0 ms. [2018-11-14 09:49:43,113][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]] [2018-11-14 09:49:43,114][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]] [2018-11-14 09:49:43,115][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 0 ms. [2018-11-14 09:49:43,115][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,115][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:43,116][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms. [2018-11-14 09:49:43,117][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 1 ms. [2018-11-14 09:49:43,117][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], err=null] [2018-11-14 09:49:43,117][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,117][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][MvccProcessorImpl] Initialize local node as mvcc coordinator [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103] [2018-11-14 09:49:43,118][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000 [2018-11-14 09:49:43,118][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983102, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], durationFromInit=10] [2018-11-14 09:49:43,119][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], crd=true] [2018-11-14 09:49:43,120][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=0], force=false, evt=NODE_JOINED, node=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,121][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,121][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] Data Regions Configured: [2018-11-14 09:49:43,121][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,122][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][TcpDiscoveryMultiThreadedTest0] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest0 >>> Local node [ID=3DC77D76-35D6-4BAF-B057-C1ECFCF00000, order=1, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10800 TCP:45180 TCP:47500 [2018-11-14 09:49:43,122][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][GridDiscoveryManager] Topology snapshot [ver=1, locNode=3dc77d76, servers=1, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,123][INFO ][test-runner-#98246%tcp.TcpDiscoveryMultiThreadedTest%][root] Node started with the following configuration [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@2ea190ae], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,127][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, customEvt=DynamicCacheChangeBatch [id=7e5f2a11761-41c60f01-3662-42a0-a5de-7fb0d43f548b, reqs=ArrayList [DynamicCacheChangeRequest [cacheName=default, hasCfg=true, nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, clientStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions [startCaches=[default], stopCaches=null, startGrps=[default], stopGrps=[], resetParts=null, stateChangeRequest=null], startCaches=false], allowMerge=false] [2018-11-14 09:49:43,128][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Updating caches registry performed in 0 ms. [2018-11-14 09:49:43,132][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,132][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Caches starting performed in 4 ms. [2018-11-14 09:49:43,133][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity initialization for started caches performed in 0 ms. [2018-11-14 09:49:43,133][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,133][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]] [2018-11-14 09:49:43,134][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,142][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [2018-11-14 09:49:43,142][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 0 ms. [2018-11-14 09:49:43,143][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,143][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:43,144][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:43,144][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:43,144][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], err=null] [2018-11-14 09:49:43,145][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000 [2018-11-14 09:49:43,146][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983102, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], durationFromInit=20] [2018-11-14 09:49:43,146][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true] [2018-11-14 09:49:43,147][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, node=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,152][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,152][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Config URL: n/a [2018-11-14 09:49:43,152][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest1, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@f49805a, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@14ff0f8a[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@2401dc37, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@2c4b0168, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@49b13ff7, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,157][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,159][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Config URL: n/a [2018-11-14 09:49:43,159][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,159][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Config URL: n/a [2018-11-14 09:49:43,155][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,160][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Config URL: n/a [2018-11-14 09:49:43,160][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest3, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@48bbb556, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@32075b16[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@174bcf36, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@174844c2, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@53eab56f, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,166][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Daemon mode: off [2018-11-14 09:49:43,166][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,166][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] OS user: teamcity [2018-11-14 09:49:43,159][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Daemon mode: off [2018-11-14 09:49:43,166][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,167][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] OS user: teamcity [2018-11-14 09:49:43,159][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=ab7119de-eeba-4180-b532-10ed81e00004, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@7eff9707, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@3ec1900b[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@2bf8c034, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@4f8b4b67, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@6e6ba88c, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,172][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Daemon mode: off [2018-11-14 09:49:43,173][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,173][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS user: teamcity [2018-11-14 09:49:43,167][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] PID: 18275 [2018-11-14 09:49:43,166][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] PID: 18275 [2018-11-14 09:49:43,161][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,173][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Config URL: n/a [2018-11-14 09:49:43,160][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@4c8fa3f, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@3bce4c08[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@59803989, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@117700ff, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@3d33128a, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,174][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@317d6aa5, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@6f9832c6[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@167ab1e4, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5f33f297, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@43edabcb, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,186][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Daemon mode: off [2018-11-14 09:49:43,186][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,187][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS user: teamcity [2018-11-14 09:49:43,173][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,187][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,173][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,187][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,188][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM total memory: 1.8GB [2018-11-14 09:49:43,188][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,173][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] PID: 18275 [2018-11-14 09:49:43,188][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,188][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,187][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM total memory: 1.8GB [2018-11-14 09:49:43,187][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] PID: 18275 [2018-11-14 09:49:43,180][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Daemon mode: off [2018-11-14 09:49:43,189][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,189][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] OS user: teamcity [2018-11-14 09:49:43,189][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,189][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,188][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,188][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,188][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,193][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,193][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,190][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,193][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,189][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM total memory: 1.8GB [2018-11-14 09:49:43,194][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,189][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] PID: 18275 [2018-11-14 09:49:43,194][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,194][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,194][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,193][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,193][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM total memory: 1.8GB [2018-11-14 09:49:43,198][WARN ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,197][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,194][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,194][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,201][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,201][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,198][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,198][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,202][WARN ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,202][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,202][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM total memory: 1.8GB [2018-11-14 09:49:43,201][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,203][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,203][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,203][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,202][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,204][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,203][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,204][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,204][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,205][INFO ][node-restart-3][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,205][INFO ][node-restart-1][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,208][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:43,204][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,212][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,212][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,212][WARN ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,209][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,209][INFO ][node-restart-1][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,207][INFO ][node-restart-3][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,213][INFO ][node-restart-3][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,214][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:43,214][INFO ][node-restart-3][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,207][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,213][INFO ][node-restart-1][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45181, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,213][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,215][WARN ][node-restart-1][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,215][INFO ][node-restart-3][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45182, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,215][INFO ][node-restart-2][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,214][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,216][INFO ][node-restart-2][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,217][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,217][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,217][INFO ][node-restart-2][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,215][WARN ][node-restart-3][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,217][WARN ][node-restart-1][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,217][INFO ][node-restart-5][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,218][INFO ][node-restart-5][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,218][INFO ][node-restart-5][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,218][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:43,216][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,218][INFO ][node-restart-5][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,219][INFO ][node-restart-5][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,219][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:43,219][INFO ][node-restart-5][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,219][INFO ][node-restart-5][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,219][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:43,218][INFO ][node-restart-2][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45183, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,218][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,219][WARN ][node-restart-2][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,219][INFO ][node-restart-5][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,219][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,221][INFO ][node-restart-5][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45184, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,221][WARN ][node-restart-5][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,219][WARN ][node-restart-3][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,221][WARN ][node-restart-2][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,221][INFO ][node-restart-1][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,222][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,222][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,223][INFO ][node-restart-4][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,224][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:43,224][INFO ][node-restart-4][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,224][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,224][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:43,223][INFO ][node-restart-1][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,224][INFO ][node-restart-2][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,224][INFO ][node-restart-4][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,224][INFO ][node-restart-2][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,224][INFO ][node-restart-3][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,225][INFO ][node-restart-3][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,225][INFO ][node-restart-1][ClientListenerProcessor] Client connector processor has started on TCP port 10801 [2018-11-14 09:49:43,226][INFO ][node-restart-2][ClientListenerProcessor] Client connector processor has started on TCP port 10802 [2018-11-14 09:49:43,225][WARN ][node-restart-5][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,226][INFO ][node-restart-4][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45185, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,226][INFO ][node-restart-3][ClientListenerProcessor] Client connector processor has started on TCP port 10803 [2018-11-14 09:49:43,226][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,227][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,227][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,226][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,227][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,226][WARN ][node-restart-4][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,228][INFO ][node-restart-2][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,227][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,228][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,228][INFO ][node-restart-5][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,228][INFO ][node-restart-1][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,228][INFO ][node-restart-5][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,229][INFO ][node-restart-3][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,229][INFO ][node-restart-3][TcpDiscoverySpi] Successfully bound to TCP port [port=47503, localHost=/127.0.0.1, locNodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:43,228][INFO ][node-restart-2][TcpDiscoverySpi] Successfully bound to TCP port [port=47501, localHost=/127.0.0.1, locNodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,229][WARN ][node-restart-4][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,229][INFO ][node-restart-1][TcpDiscoverySpi] Successfully bound to TCP port [port=47502, localHost=/127.0.0.1, locNodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001] [2018-11-14 09:49:43,231][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,231][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=58456] [2018-11-14 09:49:43,342][INFO ][node-restart-4][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,342][INFO ][node-restart-5][ClientListenerProcessor] Client connector processor has started on TCP port 10804 [2018-11-14 09:49:43,342][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=40564] [2018-11-14 09:49:43,343][INFO ][node-restart-4][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,343][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=40564] [2018-11-14 09:49:43,343][INFO ][tcp-disco-sock-reader-#26403%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:40564, rmtPort=40564] [2018-11-14 09:49:43,344][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=58456] [2018-11-14 09:49:43,344][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,345][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,345][INFO ][tcp-disco-sock-reader-#26402%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:58456, rmtPort=58456] [2018-11-14 09:49:43,345][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=51816] [2018-11-14 09:49:43,346][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=51816] [2018-11-14 09:49:43,346][INFO ][node-restart-5][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,346][INFO ][node-restart-4][ClientListenerProcessor] Client connector processor has started on TCP port 10805 [2018-11-14 09:49:43,346][INFO ][tcp-disco-sock-reader-#26404%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:51816, rmtPort=51816] [2018-11-14 09:49:43,347][INFO ][node-restart-5][TcpDiscoverySpi] Successfully bound to TCP port [port=47504, localHost=/127.0.0.1, locNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,347][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,347][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,348][INFO ][node-restart-4][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,348][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36737] [2018-11-14 09:49:43,348][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=42831] [2018-11-14 09:49:43,349][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=42831] [2018-11-14 09:49:43,349][INFO ][tcp-disco-sock-reader-#26409%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:42831, rmtPort=42831] [2018-11-14 09:49:43,348][INFO ][tcp-disco-sock-reader-#26402%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:58456, rmtPort=58456 [2018-11-14 09:49:43,348][INFO ][tcp-disco-sock-reader-#26403%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:40564, rmtPort=40564 [2018-11-14 09:49:43,349][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36737] [2018-11-14 09:49:43,349][INFO ][node-restart-4][TcpDiscoverySpi] Successfully bound to TCP port [port=47505, localHost=/127.0.0.1, locNodeId=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:43,349][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:43,349][INFO ][tcp-disco-sock-reader-#26404%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:51816, rmtPort=51816 [2018-11-14 09:49:43,350][INFO ][tcp-disco-sock-reader-#26409%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:42831, rmtPort=42831 [2018-11-14 09:49:43,350][INFO ][tcp-disco-sock-reader-#26408%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:36737, rmtPort=36737] [2018-11-14 09:49:43,353][INFO ][tcp-disco-sock-reader-#26408%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:36737, rmtPort=36737 [2018-11-14 09:49:43,353][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=0, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,353][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60572] [2018-11-14 09:49:43,354][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60572] [2018-11-14 09:49:43,354][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=49057] [2018-11-14 09:49:43,353][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36051] [2018-11-14 09:49:43,355][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36051] [2018-11-14 09:49:43,354][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=49057] [2018-11-14 09:49:43,354][INFO ][tcp-disco-sock-reader-#26411%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60572, rmtPort=60572] [2018-11-14 09:49:43,355][INFO ][tcp-disco-sock-reader-#26412%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:49057, rmtPort=49057] [2018-11-14 09:49:43,355][INFO ][tcp-disco-sock-reader-#26413%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:36051, rmtPort=36051] [2018-11-14 09:49:43,357][INFO ][tcp-disco-sock-reader-#26411%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:60572, rmtPort=60572 [2018-11-14 09:49:43,357][INFO ][tcp-disco-sock-reader-#26413%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:36051, rmtPort=36051 [2018-11-14 09:49:43,357][INFO ][tcp-disco-srvr-#26400%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=44736] [2018-11-14 09:49:43,358][INFO ][tcp-disco-srvr-#26400%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=44736] [2018-11-14 09:49:43,358][INFO ][tcp-disco-sock-reader-#26414%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:44736, rmtPort=44736] [2018-11-14 09:49:43,358][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,359][INFO ][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,359][INFO ][tcp-disco-sock-reader-#26414%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:44736, rmtPort=44736 [2018-11-14 09:49:43,360][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60315] [2018-11-14 09:49:43,360][INFO ][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60315] [2018-11-14 09:49:43,360][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=f3cd78d0-c516-4ff3-8c99-885d58e00001] [2018-11-14 09:49:43,360][INFO ][tcp-disco-sock-reader-#26415%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60315, rmtPort=60315] [2018-11-14 09:49:43,360][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=45358] [2018-11-14 09:49:43,361][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=45358] [2018-11-14 09:49:43,361][INFO ][tcp-disco-sock-reader-#26416%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:45358, rmtPort=45358] [2018-11-14 09:49:43,362][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,362][INFO ][tcp-disco-sock-reader-#26415%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:60315, rmtPort=60315 [2018-11-14 09:49:43,362][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=47000] [2018-11-14 09:49:43,363][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=47000] [2018-11-14 09:49:43,363][INFO ][tcp-disco-sock-reader-#26417%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:47000, rmtPort=47000] [2018-11-14 09:49:43,363][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,363][INFO ][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, intOrder=3, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,364][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=2, locNode=3dc77d76, servers=2, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,364][INFO ][tcp-disco-sock-reader-#26417%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:47000, rmtPort=47000 [2018-11-14 09:49:43,365][INFO ][tcp-disco-sock-reader-#26416%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:45358, rmtPort=45358 [2018-11-14 09:49:43,365][INFO ][tcp-disco-srvr-#26400%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=42950] [2018-11-14 09:49:43,365][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,365][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:43,366][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,365][INFO ][tcp-disco-srvr-#26400%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=42950] [2018-11-14 09:49:43,366][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]]] [2018-11-14 09:49:43,367][INFO ][tcp-disco-sock-reader-#26418%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:42950, rmtPort=42950] [2018-11-14 09:49:43,367][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,368][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], crd=true] [2018-11-14 09:49:43,369][INFO ][tcp-disco-msg-worker-#26394%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,370][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=58758] [2018-11-14 09:49:43,370][INFO ][disco-notifier-worker-#98458%tcp.TcpDiscoveryMultiThreadedTest3%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,373][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=58758] [2018-11-14 09:49:43,373][INFO ][tcp-disco-sock-reader-#26420%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:58758, rmtPort=58758] [2018-11-14 09:49:43,373][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,374][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,374][INFO ][tcp-disco-msg-worker-#26394%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=0, intOrder=4, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,374][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,374][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,376][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=3, locNode=2f28edb7, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,375][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=44797] [2018-11-14 09:49:43,376][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=44797] [2018-11-14 09:49:43,375][INFO ][tcp-disco-sock-reader-#26420%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:58758, rmtPort=58758 [2018-11-14 09:49:43,375][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=3, locNode=3dc77d76, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,377][INFO ][tcp-disco-sock-reader-#26421%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:44797, rmtPort=44797] [2018-11-14 09:49:43,376][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][msg] Components activation performed in 2 ms. [2018-11-14 09:49:43,381][INFO ][sys-#98480%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,381][INFO ][sys-#98481%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,381][INFO ][tcp-disco-msg-worker-#26397%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,381][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Starting caches on local join performed in 3 ms. [2018-11-14 09:49:43,382][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=59591] [2018-11-14 09:49:43,382][INFO ][disco-notifier-worker-#98455%tcp.TcpDiscoveryMultiThreadedTest2%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,382][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=59591] [2018-11-14 09:49:43,382][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]] [2018-11-14 09:49:43,384][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,383][INFO ][tcp-disco-sock-reader-#26423%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:59591, rmtPort=59591] [2018-11-14 09:49:43,385][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,384][INFO ][grid-nio-worker-tcp-comm-0-#98270%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36512] [2018-11-14 09:49:43,385][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][msg] Components activation performed in 1 ms. [2018-11-14 09:49:43,385][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,386][INFO ][grid-nio-worker-tcp-comm-0-#98405%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36512, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,386][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,386][INFO ][tcp-disco-msg-worker-#26397%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=0, intOrder=5, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,388][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=4, locNode=2f28edb7, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,387][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=true] [2018-11-14 09:49:43,387][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,386][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:43,386][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=4, locNode=3dc77d76, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,390][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], crd=false] [2018-11-14 09:49:43,389][INFO ][sys-#98488%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,389][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=4, locNode=654c3cdf, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,389][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=2, minorTopVer=0]] [2018-11-14 09:49:43,389][INFO ][sys-#98487%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,388][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36815] [2018-11-14 09:49:43,388][INFO ][tcp-disco-sock-reader-#26423%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:59591, rmtPort=59591 [2018-11-14 09:49:43,392][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36815] [2018-11-14 09:49:43,391][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Starting caches on local join performed in 6 ms. [2018-11-14 09:49:43,391][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=3, minorTopVer=0], evt=NODE_JOINED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, evtNodeClient=false] [2018-11-14 09:49:43,392][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]] [2018-11-14 09:49:43,392][INFO ][tcp-disco-sock-reader-#26424%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:36815, rmtPort=36815] [2018-11-14 09:49:43,393][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, wait for message [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,394][INFO ][grid-nio-worker-tcp-comm-1-#98271%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36514] [2018-11-14 09:49:43,394][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:43,394][INFO ][grid-nio-worker-tcp-comm-0-#98412%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36514, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,395][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, wait for message [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001] [2018-11-14 09:49:43,395][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 4 ms. [2018-11-14 09:49:43,395][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:43,395][INFO ][sys-#98289%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, received message [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, msgVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], process=true, awaited=2] [2018-11-14 09:49:43,396][INFO ][tcp-disco-msg-worker-#26405%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,396][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], crd=false] [2018-11-14 09:49:43,397][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33331] [2018-11-14 09:49:43,396][INFO ][sys-#98289%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Fast replied to single message [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], discoEvt=null, nodeId=654c3cdf, evt=NODE_JOINED], nodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,397][INFO ][disco-notifier-worker-#98456%tcp.TcpDiscoveryMultiThreadedTest1%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,397][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33331] [2018-11-14 09:49:43,398][WARN ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=85243MB, available=96649MB] [2018-11-14 09:49:43,398][INFO ][tcp-disco-sock-reader-#26425%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:33331, rmtPort=33331] [2018-11-14 09:49:43,399][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,400][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,400][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,400][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,401][INFO ][tcp-disco-msg-worker-#26405%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=0, intOrder=6, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,401][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][msg] Components activation performed in 1 ms. [2018-11-14 09:49:43,400][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=5, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,402][INFO ][tcp-disco-srvr-#26410%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33860] [2018-11-14 09:49:43,402][INFO ][tcp-disco-sock-reader-#26425%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:33331, rmtPort=33331 [2018-11-14 09:49:43,401][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=5, locNode=2f28edb7, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,404][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,403][INFO ][tcp-disco-srvr-#26410%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33860] [2018-11-14 09:49:43,405][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=5, locNode=654c3cdf, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,404][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,405][INFO ][sys-#98494%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,405][INFO ][tcp-disco-sock-reader-#26427%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:33860, rmtPort=33860] [2018-11-14 09:49:43,406][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=5, locNode=f3cd78d0, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,407][INFO ][sys-#98495%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,407][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Starting caches on local join performed in 5 ms. [2018-11-14 09:49:43,407][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,408][INFO ][grid-nio-worker-tcp-comm-2-#98272%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36516] [2018-11-14 09:49:43,409][INFO ][grid-nio-worker-tcp-comm-0-#98400%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36516, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,409][INFO ][disco-notifier-worker-#98471%tcp.TcpDiscoveryMultiThreadedTest5%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,409][INFO ][tcp-disco-msg-worker-#26407%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,409][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, received message [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, msgVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], process=true, awaited=1] [2018-11-14 09:49:43,409][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:43,410][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,410][WARN ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=106554MB, available=96649MB] [2018-11-14 09:49:43,410][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=45752] [2018-11-14 09:49:43,411][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=false] [2018-11-14 09:49:43,412][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=45752] [2018-11-14 09:49:43,412][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,412][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,413][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:43,412][INFO ][tcp-disco-sock-reader-#26429%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:45752, rmtPort=45752] [2018-11-14 09:49:43,413][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,413][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,414][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 1 ms. [2018-11-14 09:49:43,414][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][msg] Components activation performed in 1 ms. [2018-11-14 09:49:43,415][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:43,415][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,416][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=6, locNode=3dc77d76, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,416][INFO ][disco-notifier-worker-#98474%tcp.TcpDiscoveryMultiThreadedTest4%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,415][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:43,417][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 2 ms. [2018-11-14 09:49:43,417][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=127865MB, available=96649MB] [2018-11-14 09:49:43,415][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,420][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=6, locNode=f3cd78d0, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,420][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,415][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,415][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,421][INFO ][disco-event-worker-#98482%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=6, locNode=654c3cdf, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,420][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,420][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,417][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], err=null] [2018-11-14 09:49:43,416][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,423][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,422][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCacheProcessor] Starting caches on local join performed in 8 ms. [2018-11-14 09:49:43,422][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,421][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=6, locNode=2f28edb7, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,424][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=3, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:43,424][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2018-11-14 09:49:43,423][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=6, locNode=4ce119dc, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,425][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Merge exchange future on finish stop [curFut=AffinityTopologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nextFutVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2018-11-14 09:49:43,426][INFO ][grid-nio-worker-tcp-comm-3-#98273%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36518] [2018-11-14 09:49:43,426][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,427][INFO ][grid-nio-worker-tcp-comm-0-#98424%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36518, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,427][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,427][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:43,428][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false] [2018-11-14 09:49:43,427][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][msg] Components activation performed in 0 ms. [2018-11-14 09:49:43,428][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,431][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], durationFromInit=60] [2018-11-14 09:49:43,431][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,431][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:43,433][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,433][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,432][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Fast replied to single message [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], discoEvt=null, nodeId=f3cd78d0, evt=NODE_JOINED], nodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001] [2018-11-14 09:49:43,434][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2018-11-14 09:49:43,433][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity initialization on local join performed in 7 ms. [2018-11-14 09:49:43,435][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 13 ms. [2018-11-14 09:49:43,434][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=3, minorTopVer=0], evt=NODE_JOINED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, evtNodeClient=false] [2018-11-14 09:49:43,436][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,436][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], err=null] [2018-11-14 09:49:43,436][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:43,437][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,437][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity initialization on local join performed in 9 ms. [2018-11-14 09:49:43,438][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 10 ms. [2018-11-14 09:49:43,438][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002 [2018-11-14 09:49:43,439][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002 [2018-11-14 09:49:43,437][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish stop [curFut=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], nextFutVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2018-11-14 09:49:43,440][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,440][INFO ][sys-#98500%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983412, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:43,439][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,441][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], err=null] [2018-11-14 09:49:43,440][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,442][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,442][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,443][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:43,443][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:43,443][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity initialization on local join performed in 3 ms. [2018-11-14 09:49:43,444][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 10 ms. [2018-11-14 09:49:43,444][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, node=f3cd78d0-c516-4ff3-8c99-885d58e00001] [2018-11-14 09:49:43,443][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983412, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:43,445][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,445][INFO ][sys-#98521%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,445][INFO ][sys-#98520%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,445][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,447][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Starting caches on local join performed in 19 ms. [2018-11-14 09:49:43,447][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,448][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,447][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], err=null] [2018-11-14 09:49:43,449][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,449][INFO ][grid-nio-worker-tcp-comm-0-#98270%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36520] [2018-11-14 09:49:43,449][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:43,450][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:43,450][INFO ][grid-nio-worker-tcp-comm-0-#98440%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36520, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,450][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_JOINED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:43,450][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983412, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], durationFromInit=71] [2018-11-14 09:49:43,451][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,450][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:43,452][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], crd=false] [2018-11-14 09:49:43,452][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,453][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]] [2018-11-14 09:49:43,453][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]] [2018-11-14 09:49:43,453][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]] [2018-11-14 09:49:43,454][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,454][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]]] [2018-11-14 09:49:43,454][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,456][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, allReceived=false] [2018-11-14 09:49:43,456][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,456][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:43,455][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,457][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,457][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false] [2018-11-14 09:49:43,457][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:43,458][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,459][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false] [2018-11-14 09:49:43,458][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=false] [2018-11-14 09:49:43,459][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=false] [2018-11-14 09:49:43,459][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], crd=true] [2018-11-14 09:49:43,460][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, allReceived=true] [2018-11-14 09:49:43,461][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2018-11-14 09:49:43,461][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, evtNodeClient=false] [2018-11-14 09:49:43,461][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, message received [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:43,462][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 1 ms. [2018-11-14 09:49:43,462][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,463][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,464][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:43,464][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,465][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:43,466][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:43,467][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 1 ms. [2018-11-14 09:49:43,467][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,468][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,468][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,468][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,468][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 2 ms. [2018-11-14 09:49:43,468][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,469][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,469][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, evtNodeClient=false] [2018-11-14 09:49:43,468][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,470][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,469][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, evtNodeClient=false] [2018-11-14 09:49:43,470][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, evtNodeClient=false] [2018-11-14 09:49:43,471][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:43,471][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,472][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 0 ms. [2018-11-14 09:49:43,473][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 5 ms. [2018-11-14 09:49:43,472][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,472][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 2 ms. [2018-11-14 09:49:43,474][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,474][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 6 ms. [2018-11-14 09:49:43,474][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,474][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 6 ms. [2018-11-14 09:49:43,474][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,474][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,476][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]] [2018-11-14 09:49:43,476][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,475][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,475][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,477][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,477][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, evtNodeClient=false] [2018-11-14 09:49:43,479][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][CacheAffinitySharedManager] Affinity initialization on local join performed in 2 ms. [2018-11-14 09:49:43,479][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:43,479][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:43,479][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,480][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,480][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=ab7119de-eeba-4180-b532-10ed81e00004 [2018-11-14 09:49:43,480][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=ab7119de-eeba-4180-b532-10ed81e00004 [2018-11-14 09:49:43,481][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ab7119de-eeba-4180-b532-10ed81e00004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983422, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=61] [2018-11-14 09:49:43,478][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,478][INFO ][sys-#98485%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=40] [2018-11-14 09:49:43,478][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,482][INFO ][sys-#98477%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=30] [2018-11-14 09:49:43,481][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity initialization on local join performed in 2 ms. [2018-11-14 09:49:43,484][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 8 ms. [2018-11-14 09:49:43,481][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:43,480][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,485][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,485][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,483][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,483][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,487][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,487][INFO ][sys-#98518%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,486][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], err=null] [2018-11-14 09:49:43,486][INFO ][exchange-worker-#98483%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,489][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,489][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,488][INFO ][utility-#98532%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,488][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,488][INFO ][utility-#98531%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,488][INFO ][sys-#98481%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,487][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=40] [2018-11-14 09:49:43,487][INFO ][utility-#98530%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=ab7119de-eeba-4180-b532-10ed81e00004, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,491][INFO ][utility-#98531%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,490][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:43,493][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:43,490][INFO ][sys-#98495%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,490][INFO ][utility-#98535%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,489][INFO ][utility-#98534%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,489][INFO ][sys-#98488%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,489][INFO ][utility-#98533%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,494][INFO ][utility-#98538%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=11 ms] [2018-11-14 09:49:43,494][INFO ][utility-#98537%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=f3cd78d0-c516-4ff3-8c99-885d58e00001, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,494][INFO ][utility-#98535%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,494][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,494][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=6, minorTopVer=0], force=false, evt=NODE_JOINED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,493][INFO ][sys-#98522%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983422, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], durationFromInit=81] [2018-11-14 09:49:43,492][INFO ][utility-#98531%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=177, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,497][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,497][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,496][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Data Regions Configured: [2018-11-14 09:49:43,496][INFO ][utility-#98535%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=154, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,495][INFO ][utility-#98538%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,495][INFO ][utility-#98533%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,500][INFO ][utility-#98538%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=184, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,499][INFO ][utility-#98540%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=4ce119dc-4ac4-43db-b3ca-aba085b00005, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,500][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[6, 17, 23, 27, 36-37, 44-45, 51, 58, 60, 67, 73, 75-76, 83, 90, 94-95, 98, 105, 112, 117-118, 125, 131, 136, 138-140, 144, 156, 170, 173-174, 190, 192-193, 200, 204, 208, 219, 230, 240, 244, 249, 253-255, 257, 262, 265, 276, 288, 294, 296, 299, 318-319, 321-323, 325, 328, 333, 338-339, 346, 348, 351, 362, 369, 374, 389, 400, 406, 408-409, 420, 424-425, 458, 466, 473-474, 477, 480, 488, 497, 503, 509, 517, 520-521, 523, 527, 538, 541, 543, 547-548, 563, 569, 579, 585, 596, 601, 604, 611, 621-622, 630, 644, 650, 655, 657, 662, 671, 676-677, 680, 682, 688, 693, 702, 712-713, 716, 720, 722, 724-725, 734, 736, 747-748, 750, 754, 768-769, 771, 778, 780, 788, 791, 793, 795, 805-806, 812, 817, 828, 832, 847, 854, 859-860, 885, 887, 889, 895-896, 903, 908, 916, 919, 924, 930, 933, 946, 953, 955, 970, 975, 979, 982, 987, 994, 996, 1004, 1009, 1012-1013, 1017], histPartitions=[]] [2018-11-14 09:49:43,499][INFO ][sys-#98498%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,499][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,502][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest4 >>> Local node [ID=AB7119DE-EEBA-4180-B532-10ED81E00004, order=6, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10805 TCP:45185 TCP:47505 [2018-11-14 09:49:43,503][INFO ][node-restart-4][GridDiscoveryManager] Topology snapshot [ver=6, locNode=ab7119de, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,503][INFO ][node-restart-4][root] Node started with the following configuration [id=ab7119de-eeba-4180-b532-10ed81e00004, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@512e0e24], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,507][INFO ][node-restart-4][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,498][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Data Regions Configured: [2018-11-14 09:49:43,503][INFO ][sys-#98528%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=11 ms] [2018-11-14 09:49:43,503][INFO ][sys-#98504%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,502][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=ab7119de-eeba-4180-b532-10ed81e00004, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,502][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,502][INFO ][sys-#98529%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[18-19, 22, 26, 29, 32, 39-40, 42, 56-57, 59, 61-63, 69, 71, 79-80, 86, 88, 101, 107, 109, 114, 121, 130, 137, 141, 153, 155, 157, 161, 166, 175, 177-179, 184, 187-188, 194, 198, 209, 211, 213, 215, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 330, 337, 341, 343, 349, 352, 363, 373, 382-383, 392, 401-402, 407, 422, 426, 429-430, 432, 440, 443-444, 467, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 613, 620, 641-642, 678, 686, 695-696, 698, 700-701, 706, 710, 718-719, 730, 741-742, 744, 752, 757, 762, 772-774, 776, 802, 811, 818, 821, 823, 827, 829, 834, 869, 872, 892, 905, 912, 914, 920, 925, 931, 939, 943, 948, 956-957, 959, 967, 969, 976-977, 986, 989, 993, 995, 1015, 1022], histPartitions=[]] [2018-11-14 09:49:43,502][INFO ][sys-#98503%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[1, 4-5, 7-8, 15, 24, 31, 50, 54-55, 64, 70, 78, 85, 108, 110-111, 115, 124, 129, 134, 142, 148-149, 159, 171-172, 181, 195-196, 205-206, 217-218, 220, 222, 224, 237, 242, 266-267, 269, 275, 284, 291, 295, 311, 317, 329, 331, 366, 378, 380-381, 385, 390, 393, 395, 399, 412, 417, 428, 435-436, 462, 465, 468, 471, 478, 494, 500, 522, 525, 553, 557, 564, 571, 582-584, 591, 598, 605-606, 608-609, 614, 616, 619, 628, 633, 635, 648-649, 654, 656, 707, 709, 723, 728-729, 731, 733, 735, 738, 740, 756, 761, 763, 766, 770, 779, 784, 787, 790, 796-797, 801, 804, 809, 813, 820, 840, 845, 851, 855, 863, 865-866, 874, 877, 879, 894, 897, 906, 909, 921, 937-938, 942, 944, 947, 951-952, 961-962, 968, 980-981, 983, 1002, 1014, 1020], histPartitions=[]] [2018-11-14 09:49:43,501][INFO ][sys-#98511%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,512][INFO ][sys-#98511%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,501][INFO ][sys-#98289%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=f3cd78d0-c516-4ff3-8c99-885d58e00001, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,501][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,501][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,513][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Data Regions Configured: [2018-11-14 09:49:43,514][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,514][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest3 >>> Local node [ID=2F28EDB7-BF70-4CA2-9CB7-D18C1F500003, order=2, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10803 TCP:45182 TCP:47503 [2018-11-14 09:49:43,515][INFO ][node-restart-3][GridDiscoveryManager] Topology snapshot [ver=2, locNode=2f28edb7, servers=2, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,515][INFO ][node-restart-3][root] Node started with the following configuration [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@268bc636], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,516][INFO ][node-restart-3][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,500][INFO ][utility-#98541%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,500][INFO ][utility-#98533%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=179, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,513][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Data Regions Configured: [2018-11-14 09:49:43,517][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,517][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest1 >>> Local node [ID=F3CD78D0-C516-4FF3-8C99-885D58E00001, order=4, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10801 TCP:45181 TCP:47502 [2018-11-14 09:49:43,518][INFO ][node-restart-1][GridDiscoveryManager] Topology snapshot [ver=4, locNode=f3cd78d0, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,518][INFO ][node-restart-1][root] Node started with the following configuration [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@2fae37ce], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,519][INFO ][node-restart-1][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,509][INFO ][disco-notifier-worker-#98282%tcp.TcpDiscoveryMultiThreadedTest0%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,508][INFO ][sys-#98504%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,508][INFO ][sys-#98528%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,508][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,524][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest2 >>> Local node [ID=654C3CDF-95B8-4F67-8F06-1C4AB7E00002, order=3, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10802 TCP:45183 TCP:47501 [2018-11-14 09:49:43,525][INFO ][node-restart-2][GridDiscoveryManager] Topology snapshot [ver=3, locNode=654c3cdf, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,525][INFO ][node-restart-2][root] Node started with the following configuration [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@792a1640], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,526][INFO ][node-restart-2][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,519][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=20 ms] [2018-11-14 09:49:43,527][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,518][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,517][INFO ][utility-#98541%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,517][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0, 12, 14, 46, 48-49, 65-66, 74, 77, 84, 92, 96, 99, 103-104, 113, 120, 123, 133, 143, 145-147, 162-163, 167, 169, 180, 183, 201, 203, 212, 221, 223, 227, 231-233, 246, 252, 258-259, 272, 278, 287, 298, 303, 308-309, 320, 326-327, 332, 340, 342, 344, 347, 350, 354-357, 361, 379, 386, 388, 413, 419, 423, 431, 447, 453-454, 459, 463-464, 469, 485, 487, 490-492, 496, 498, 502, 532-533, 536, 539, 542, 545, 549, 552, 555, 561, 570, 576, 581, 597, 612, 617-618, 624, 627, 637, 639, 643, 651, 658, 660-661, 663, 665, 668, 672-675, 679, 687, 689, 691, 699, 704-705, 717, 737, 743, 745-746, 749, 751, 759, 767, 775, 777, 807-808, 814, 819, 822, 830, 843, 846, 848-849, 853, 856, 862, 864, 873, 875-876, 880, 890-891, 899, 911, 915, 922, 927, 936, 940-941, 949, 954, 964, 966, 972, 984, 990-991, 1003, 1005-1008, 1010], histPartitions=[]] [2018-11-14 09:49:43,530][INFO ][utility-#98541%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=165, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,531][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[9, 16, 21, 28, 30, 33-35, 41, 43, 47, 52, 68, 81, 91, 93, 102, 106, 116, 119, 126, 128, 135, 154, 185, 197, 199, 202, 207, 214, 216, 226, 241, 247, 261, 268, 271, 274, 279-280, 282-283, 289, 302, 304, 335, 358-359, 365, 367-368, 375, 377, 387, 391, 394, 397-398, 403-404, 414-416, 418, 433, 437, 441-442, 448, 460-461, 472, 475-476, 479, 481, 484, 489, 493, 495, 510-511, 513-514, 518, 528, 537, 540, 546, 551, 558, 560, 562, 568, 572, 574, 586, 592-593, 595, 603, 607, 610, 626, 629, 632, 634, 640, 645, 652, 659, 664, 667, 681, 690, 692, 694, 703, 708, 714, 721, 753, 760, 765, 783, 786, 794, 798, 800, 803, 815, 824, 831, 833, 835, 837, 839, 841, 858, 861, 868, 870, 881-883, 888, 901, 904, 913, 917-918, 923, 929, 932, 934-935, 945, 958, 960, 963, 978, 992, 997-998, 1000], histPartitions=[]] [2018-11-14 09:49:43,532][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=4ce119dc-4ac4-43db-b3ca-aba085b00005, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], topic=0] [2018-11-14 09:49:43,533][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,533][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Data Regions Configured: [2018-11-14 09:49:43,533][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,534][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest5 >>> Local node [ID=4CE119DC-4AC4-43DB-B3CA-ABA085B00005, order=5, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10804 TCP:45184 TCP:47504 [2018-11-14 09:49:43,534][INFO ][sys-#98526%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], progress=1/1, time=10 ms] [2018-11-14 09:49:43,535][INFO ][sys-#98526%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,534][INFO ][node-restart-5][GridDiscoveryManager] Topology snapshot [ver=5, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,536][INFO ][node-restart-5][root] Node started with the following configuration [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@625b382a], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,537][INFO ][node-restart-5][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,608][WARN ][node-restart-2][TcpDiscoverySpi] Simulating node failure: 654c3cdf-95b8-4f67-8f06-1c4ab7e00002 [2018-11-14 09:49:43,608][ERROR][tcp-disco-srvr-#26400%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:43,610][INFO ][tcp-disco-sock-reader-#26418%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:42950, rmtPort=42950 [2018-11-14 09:49:43,611][ERROR][tcp-disco-msg-worker-#26394%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:43,614][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:43,615][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:43,616][INFO ][node-restart-2][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983412, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=877f2a11761-654c3cdf-95b8-4f67-8f06-1c4ab7e00002, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:43,616][WARN ][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, id=4a9f2a11761-2f28edb7-bf70-4ca2-9cb7-d18c1f500003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=6, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, id=4a9f2a11761-2f28edb7-bf70-4ca2-9cb7-d18c1f500003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=6, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, order=3, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:43,617][INFO ][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,618][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=41008] [2018-11-14 09:49:43,618][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=41008] [2018-11-14 09:49:43,619][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45183, failureDetectionTimeout=10000] [2018-11-14 09:49:43,619][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45183]] [2018-11-14 09:49:43,619][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 654c3cdf-95b8-4f67-8f06-1c4ab7e00002 [2018-11-14 09:49:43,620][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, res=false, time=4ms] [2018-11-14 09:49:43,620][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): 654c3cdf-95b8-4f67-8f06-1c4ab7e00002 [2018-11-14 09:49:43,620][INFO ][tcp-disco-sock-reader-#26431%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:41008, rmtPort=41008] [2018-11-14 09:49:43,620][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest2 >>> Grid uptime: 00:00:00.122 [2018-11-14 09:49:43,622][INFO ][tcp-disco-sock-reader-#26431%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Connection check done: [liveAddr=null, previousNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47501], connectingNodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:43,623][INFO ][tcp-disco-sock-reader-#26431%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=1542188983605, now=1542188983616, connCheckInterval=500] [2018-11-14 09:49:43,626][WARN ][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Local node has detected failed nodes and started cluster-wide procedure. To speed up failure detection please see 'Failure Detection' section under javadoc for 'TcpDiscoverySpi' [2018-11-14 09:49:43,626][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:43,627][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Config URL: n/a [2018-11-14 09:49:43,627][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=77abe0df-a676-4745-b311-5aa4bb000002, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@26d57a56, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@4bd95c04[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@1e9bc312, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@1fe3b3a2, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1ee52d62, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:43,633][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Daemon mode: off [2018-11-14 09:49:43,633][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:43,633][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS user: teamcity [2018-11-14 09:49:43,633][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] PID: 18275 [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM total memory: 1.8GB [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:43,634][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:43,635][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:43,637][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:43,638][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:43,638][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:43,639][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:43,641][INFO ][node-restart-2][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:43,641][INFO ][node-restart-2][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:43,641][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,641][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:43,642][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:43,642][INFO ][node-restart-2][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:43,646][INFO ][node-restart-2][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45183, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:43,647][WARN ][node-restart-2][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:43,655][WARN ][node-restart-2][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:43,655][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:43,659][INFO ][node-restart-2][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:43,659][INFO ][node-restart-2][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:43,669][INFO ][node-restart-2][ClientListenerProcessor] Client connector processor has started on TCP port 10802 [2018-11-14 09:49:43,671][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,671][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=7, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,673][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:43,673][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:43,673][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,674][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,676][INFO ][node-restart-2][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:43,677][INFO ][node-restart-2][TcpDiscoverySpi] Successfully bound to TCP port [port=47501, localHost=/127.0.0.1, locNodeId=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,679][WARN ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,679][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=7, locNode=2f28edb7, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,680][WARN ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,680][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=7, locNode=f3cd78d0, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,681][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,681][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,680][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=52985] [2018-11-14 09:49:43,681][WARN ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,684][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=7, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,685][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,685][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,680][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,687][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,683][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=52985] [2018-11-14 09:49:43,688][INFO ][tcp-disco-sock-reader-#26436%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:52985, rmtPort=52985] [2018-11-14 09:49:43,690][INFO ][tcp-disco-sock-reader-#26436%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:52985, rmtPort=52985 [2018-11-14 09:49:43,692][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,696][WARN ][disco-event-worker-#98502%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,696][INFO ][disco-event-worker-#98502%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=7, locNode=ab7119de, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,697][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,698][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,699][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]]] [2018-11-14 09:49:43,699][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,699][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=true] [2018-11-14 09:49:43,700][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]]] [2018-11-14 09:49:43,701][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,700][INFO ][tcp-disco-msg-worker-#26407%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=0, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,700][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]]] [2018-11-14 09:49:43,700][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]]] [2018-11-14 09:49:43,703][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,700][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]]] [2018-11-14 09:49:43,703][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,704][INFO ][sys-#98289%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, allReceived=false] [2018-11-14 09:49:43,702][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,702][INFO ][tcp-disco-sock-reader-#26429%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:45752, rmtPort=45752 [2018-11-14 09:49:43,705][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,702][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=39028] [2018-11-14 09:49:43,701][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,706][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=39028] [2018-11-14 09:49:43,705][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false] [2018-11-14 09:49:43,705][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=ab7119de-eeba-4180-b532-10ed81e00004, allReceived=false] [2018-11-14 09:49:43,704][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false] [2018-11-14 09:49:43,704][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,707][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:43,708][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:43,706][INFO ][tcp-disco-sock-reader-#26437%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39028, rmtPort=39028] [2018-11-14 09:49:43,706][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false] [2018-11-14 09:49:43,708][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=7, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=true] [2018-11-14 09:49:43,709][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,709][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:43,709][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,708][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], crd=false] [2018-11-14 09:49:43,711][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:43,712][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:43,712][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,713][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:43,713][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms. [2018-11-14 09:49:43,714][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:43,714][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:43,714][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [2018-11-14 09:49:43,714][INFO ][tcp-disco-msg-worker-#26434%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,715][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,716][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=52586] [2018-11-14 09:49:43,716][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,716][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:43,716][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,717][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,717][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:43,719][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 3 ms. [2018-11-14 09:49:43,717][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0]] [2018-11-14 09:49:43,716][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=52586] [2018-11-14 09:49:43,719][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:43,719][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:43,720][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:43,719][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity applying from full message performed in 2 ms. [2018-11-14 09:49:43,719][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:43,721][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 4 ms. [2018-11-14 09:49:43,719][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [2018-11-14 09:49:43,722][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:43,722][INFO ][sys-#98527%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ab7119de-eeba-4180-b532-10ed81e00004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], durationFromInit=20] [2018-11-14 09:49:43,723][INFO ][sys-#98495%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,721][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:43,721][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:43,721][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 3 ms. [2018-11-14 09:49:43,724][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:43,725][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [2018-11-14 09:49:43,725][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,724][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [2018-11-14 09:49:43,724][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], err=null] [2018-11-14 09:49:43,727][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:43,727][INFO ][sys-#98478%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], durationFromInit=40] [2018-11-14 09:49:43,728][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], durationFromInit=40] [2018-11-14 09:49:43,724][INFO ][tcp-disco-sock-reader-#26438%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:52586, rmtPort=52586] [2018-11-14 09:49:43,723][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[default], top=AffinityTopologyVersion [topVer=7, minorTopVer=0], force=false, evt=NODE_FAILED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,729][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[default], top=AffinityTopologyVersion [topVer=7, minorTopVer=0], force=false, evt=NODE_FAILED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,728][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[default], top=AffinityTopologyVersion [topVer=7, minorTopVer=0], force=false, evt=NODE_FAILED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,730][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=27, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,731][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[1, 149, 222, 295, 317, 366, 380, 412, 428, 436, 494, 557, 564, 571, 582, 609, 633, 654, 723, 763, 784, 790, 796-797, 809, 937, 980], histPartitions=[]] [2018-11-14 09:49:43,726][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,726][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=7, minorTopVer=0], force=false, evt=NODE_FAILED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,725][INFO ][sys-#98498%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,724][INFO ][sys-#98480%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:43,733][INFO ][sys-#98492%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=654c3cdf-95b8-4f67-8f06-1c4ab7e00002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=3, intOrder=3, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], durationFromInit=52] [2018-11-14 09:49:43,732][INFO ][disco-event-worker-#98502%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,734][INFO ][disco-event-worker-#98502%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=8, locNode=ab7119de, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,735][INFO ][disco-notifier-worker-#98581%tcp.TcpDiscoveryMultiThreadedTest2%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,732][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,736][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=8, locNode=4ce119dc, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,732][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,737][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=8, locNode=f3cd78d0, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,731][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,738][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,731][INFO ][sys-#98286%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=4ce119dc-4ac4-43db-b3ca-aba085b00005, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], topic=0] [2018-11-14 09:49:43,731][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,730][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=40, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,730][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=36, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,740][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,741][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,739][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=8, locNode=3dc77d76, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,741][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,742][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,738][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,735][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=127865MB, available=96649MB] [2018-11-14 09:49:43,733][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:43,745][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=8, locNode=2f28edb7, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,733][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[default], top=AffinityTopologyVersion [topVer=7, minorTopVer=0], force=false, evt=NODE_FAILED, node=654c3cdf-95b8-4f67-8f06-1c4ab7e00002] [2018-11-14 09:49:43,743][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,742][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], progress=1/1, time=10 ms] [2018-11-14 09:49:43,742][INFO ][sys-#98287%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], topic=0] [2018-11-14 09:49:43,742][INFO ][sys-#98501%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[15, 55, 64, 85, 111, 124, 129, 171, 196, 206, 217-218, 237, 284, 291, 311, 331, 378, 385, 393, 417, 468, 478, 525, 553, 584, 598, 616, 656, 731, 761, 787, 804, 813, 909, 921, 938, 951-952, 1014], histPartitions=[]] [2018-11-14 09:49:43,741][INFO ][sys-#98520%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], progress=1/1, time=10 ms] [2018-11-14 09:49:43,747][INFO ][sys-#98520%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], rebalanceId=2, routines=1] [2018-11-14 09:49:43,740][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=ab7119de-eeba-4180-b532-10ed81e00004, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], topic=0] [2018-11-14 09:49:43,740][INFO ][sys-#98521%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[7, 24, 31, 70, 78, 115, 172, 195, 205, 220, 224, 329, 381, 435, 471, 605-606, 608, 614, 619, 628, 648-649, 707, 729, 738, 766, 779, 820, 866, 874, 947, 962, 981, 983, 1002], histPartitions=[]] [2018-11-14 09:49:43,746][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], rebalanceId=3, routines=1] [2018-11-14 09:49:43,745][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,745][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=27, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,750][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,750][INFO ][sys-#98505%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[4-5, 8, 54, 110, 142, 159, 242, 465, 583, 591, 635, 728, 733, 740, 756, 840, 845, 851, 863, 877, 879, 906, 944, 961, 968, 1020], histPartitions=[]] [2018-11-14 09:49:43,750][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,751][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], progress=1/1, time=10 ms] [2018-11-14 09:49:43,750][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=f3cd78d0-c516-4ff3-8c99-885d58e00001, topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], topic=0] [2018-11-14 09:49:43,752][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], rebalanceId=3, routines=1] [2018-11-14 09:49:43,751][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,754][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=true] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, customEvt=null, allowMerge=true] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983402, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [2018-11-14 09:49:43,753][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]]] [2018-11-14 09:49:43,757][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,758][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,758][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false] [2018-11-14 09:49:43,756][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,759][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,760][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false] [2018-11-14 09:49:43,756][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:43,755][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,760][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][msg] Components activation performed in 5 ms. [2018-11-14 09:49:43,759][INFO ][sys-#98286%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=ab7119de-eeba-4180-b532-10ed81e00004, allReceived=false] [2018-11-14 09:49:43,757][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:43,761][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:43,762][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false] [2018-11-14 09:49:43,761][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=false] [2018-11-14 09:49:43,763][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, allReceived=false] [2018-11-14 09:49:43,763][INFO ][sys-#98287%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:43,762][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:43,764][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false] [2018-11-14 09:49:43,767][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:43,769][INFO ][sys-#98597%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:43,777][INFO ][disco-notifier-worker-#98581%tcp.TcpDiscoveryMultiThreadedTest2%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:43,777][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Starting caches on local join performed in 16 ms. [2018-11-14 09:49:43,778][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,779][INFO ][grid-nio-worker-tcp-comm-1-#98271%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36528] [2018-11-14 09:49:43,785][INFO ][grid-nio-worker-tcp-comm-0-#98568%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36528, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:43,789][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 11 ms. [2018-11-14 09:49:43,789][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=true] [2018-11-14 09:49:43,789][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,790][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:43,790][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,791][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,789][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], crd=false] [2018-11-14 09:49:43,791][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 1 ms. [2018-11-14 09:49:43,792][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:43,792][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:43,793][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:43,793][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:43,793][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:43,794][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,794][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,794][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,795][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,795][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 0 ms. [2018-11-14 09:49:43,795][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:43,795][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,796][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:43,795][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,797][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 0 ms. [2018-11-14 09:49:43,795][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,798][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:43,798][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,799][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:43,799][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:43,799][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:43,805][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 8 ms. [2018-11-14 09:49:43,806][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]] [2018-11-14 09:49:43,806][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 7 ms. [2018-11-14 09:49:43,806][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,807][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,807][INFO ][sys-#98509%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:43,806][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 8 ms. [2018-11-14 09:49:43,806][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 7 ms. [2018-11-14 09:49:43,806][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,809][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,809][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,807][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,810][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,810][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,811][INFO ][sys-#98525%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ab7119de-eeba-4180-b532-10ed81e00004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=71] [2018-11-14 09:49:43,810][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:43,812][INFO ][sys-#98499%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=81] [2018-11-14 09:49:43,812][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,811][INFO ][exchange-worker-#98510%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,811][INFO ][sys-#98515%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=71] [2018-11-14 09:49:43,813][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,815][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity initialization on local join performed in 9 ms. [2018-11-14 09:49:43,815][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 9 ms. [2018-11-14 09:49:43,818][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 2 ms. [2018-11-14 09:49:43,818][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], err=null] [2018-11-14 09:49:43,818][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:43,819][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:43,819][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:43,820][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983810, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], durationFromInit=61] [2018-11-14 09:49:43,820][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=8, minorTopVer=0], force=false, evt=NODE_JOINED, node=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:43,821][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,822][INFO ][sys-#98593%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:43,822][INFO ][utility-#98602%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=77abe0df-a676-4745-b311-5aa4bb000002, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], topic=0] [2018-11-14 09:49:43,827][INFO ][utility-#98603%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,828][INFO ][utility-#98603%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,829][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:43,829][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Data Regions Configured: [2018-11-14 09:49:43,830][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:43,830][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest2 >>> Local node [ID=77ABE0DF-A676-4745-B311-5AA4BB000002, order=8, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10802 TCP:45183 TCP:47501 [2018-11-14 09:49:43,831][INFO ][node-restart-2][GridDiscoveryManager] Topology snapshot [ver=8, locNode=77abe0df, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:43,831][INFO ][node-restart-2][root] Node started with the following configuration [id=77abe0df-a676-4745-b311-5aa4bb000002, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@62886fe1], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:43,833][INFO ][utility-#98603%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=154, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], parallelism=1] [2018-11-14 09:49:43,833][INFO ][sys-#98601%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[1, 4-5, 7-8, 15, 24, 31, 50, 54-55, 64, 70, 78, 85, 108, 110-111, 115, 124, 129, 134, 142, 148-149, 159, 171-172, 181, 195-196, 205-206, 217-218, 220, 222, 224, 237, 242, 266-267, 269, 275, 284, 291, 295, 311, 317, 329, 331, 366, 378, 380-381, 385, 390, 393, 395, 399, 412, 417, 428, 435-436, 462, 465, 468, 471, 478, 494, 500, 522, 525, 553, 557, 564, 571, 582-584, 591, 598, 605-606, 608-609, 614, 616, 619, 628, 633, 635, 648-649, 654, 656, 707, 709, 723, 728-729, 731, 733, 735, 738, 740, 756, 761, 763, 766, 770, 779, 784, 787, 790, 796-797, 801, 804, 809, 813, 820, 840, 845, 851, 855, 863, 865-866, 874, 877, 879, 894, 897, 906, 909, 921, 937-938, 942, 944, 947, 951-952, 961-962, 968, 980-981, 983, 1002, 1014, 1020], histPartitions=[]] [2018-11-14 09:49:43,834][INFO ][sys-#98287%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=77abe0df-a676-4745-b311-5aa4bb000002, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], topic=0] [2018-11-14 09:49:43,835][INFO ][utility-#98603%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:43,835][INFO ][utility-#98603%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:43,963][INFO ][tcp-disco-sock-reader-#26421%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:44797, rmtPort=44797 [2018-11-14 09:49:43,975][WARN ][node-restart-4][TcpDiscoverySpi] Simulating node failure: ab7119de-eeba-4180-b532-10ed81e00004 [2018-11-14 09:49:43,976][ERROR][tcp-disco-srvr-#26410%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:43,977][INFO ][tcp-disco-sock-reader-#26427%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:33860, rmtPort=33860 [2018-11-14 09:49:43,978][ERROR][tcp-disco-msg-worker-#26407%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:43,982][INFO ][node-restart-4][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:43,983][INFO ][node-restart-4][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:43,983][INFO ][node-restart-4][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983463, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=0a7f2a11761-ab7119de-eeba-4180-b532-10ed81e00004, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:43,985][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest4 >>> Grid uptime: 00:00:00.489 [2018-11-14 09:49:43,985][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45185, failureDetectionTimeout=10000] [2018-11-14 09:49:43,986][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45185]] [2018-11-14 09:49:43,986][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: ab7119de-eeba-4180-b532-10ed81e00004 [2018-11-14 09:49:43,986][WARN ][tcp-disco-msg-worker-#26405%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=3d203a11761-4ce119dc-4ac4-43db-b3ca-aba085b00005, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryConnectionCheckMessage [super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=3d203a11761-4ce119dc-4ac4-43db-b3ca-aba085b00005, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=ab7119de-eeba-4180-b532-10ed81e00004, order=6, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:43,986][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=ab7119de-eeba-4180-b532-10ed81e00004, res=false, time=3ms] [2018-11-14 09:49:43,988][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): ab7119de-eeba-4180-b532-10ed81e00004 [2018-11-14 09:49:43,988][INFO ][tcp-disco-msg-worker-#26405%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983697, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:43,995][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=47911] [2018-11-14 09:49:43,995][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=47911] [2018-11-14 09:49:43,995][INFO ][tcp-disco-sock-reader-#26440%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:47911, rmtPort=47911] [2018-11-14 09:49:43,996][INFO ][tcp-disco-sock-reader-#26440%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Connection check done: [liveAddr=null, previousNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47505], connectingNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:43,996][INFO ][tcp-disco-sock-reader-#26440%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=1542188983973, now=1542188983993, connCheckInterval=500] [2018-11-14 09:49:44,000][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,000][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Config URL: n/a [2018-11-14 09:49:44,001][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@7527ae0b, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@5a3fa29[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@6df49f, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@30114d75, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@6c0bf777, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,006][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Daemon mode: off [2018-11-14 09:49:44,006][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,006][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS user: teamcity [2018-11-14 09:49:44,006][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] PID: 18275 [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM total memory: 1.8GB [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,007][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,008][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,011][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,011][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,011][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,012][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,015][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,016][INFO ][node-restart-4][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,016][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,016][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,016][INFO ][node-restart-4][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,017][INFO ][node-restart-4][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45185, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,018][WARN ][node-restart-4][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,019][WARN ][node-restart-4][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,020][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,021][INFO ][node-restart-4][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,022][INFO ][node-restart-4][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,031][INFO ][node-restart-4][ClientListenerProcessor] Client connector processor has started on TCP port 10805 [2018-11-14 09:49:44,036][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,036][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=9, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,037][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,038][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,036][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,038][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,037][WARN ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,039][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=9, locNode=2f28edb7, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,039][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,040][INFO ][node-restart-4][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,040][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,041][INFO ][node-restart-4][TcpDiscoverySpi] Successfully bound to TCP port [port=47505, localHost=/127.0.0.1, locNodeId=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,042][WARN ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,043][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=9, locNode=f3cd78d0, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,043][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,043][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=44122] [2018-11-14 09:49:44,044][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,044][WARN ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,045][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=9, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,045][WARN ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,044][INFO ][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=44122] [2018-11-14 09:49:44,045][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=9, locNode=77abe0df, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,045][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,046][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=ab7119de-eeba-4180-b532-10ed81e00004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,047][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,047][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,047][INFO ][tcp-disco-sock-reader-#26445%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:44122, rmtPort=44122] [2018-11-14 09:49:44,048][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]]] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], crd=true] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]]] [2018-11-14 09:49:44,050][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,051][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,051][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], crd=false] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]]] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]]] [2018-11-14 09:49:44,049][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]]] [2018-11-14 09:49:44,053][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,052][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,052][INFO ][sys-#98286%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=9, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:44,055][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,055][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], crd=false] [2018-11-14 09:49:44,052][INFO ][tcp-disco-sock-reader-#26445%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:44122, rmtPort=44122 [2018-11-14 09:49:44,055][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=9, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=false] [2018-11-14 09:49:44,054][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,057][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], crd=false] [2018-11-14 09:49:44,054][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,056][INFO ][sys-#98287%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=9, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, allReceived=false] [2018-11-14 09:49:44,058][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,059][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], crd=false] [2018-11-14 09:49:44,059][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=9, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=true] [2018-11-14 09:49:44,059][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,060][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:44,060][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,061][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:44,062][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:44,062][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:44,063][WARN ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[7, 18-19, 22, 24, 26, 29, 31-32, 39-40, 42, 56-57, 59, 61-63, 69-71, 78-80, 86, 88, 101, 107, 109, 114-115, 121, 130, 137, 141, 153, 155, 157, 161, 166, 172, 175, 177-179, 184, 187-188, 194-195, 198, 205, 209, 211, 213, 215, 220, 224, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 329-330, 337, 341, 343, 349, 352, 363, 373, 381-383, 392, 401-402, 407, 422, 426, 429-430, 432, 435, 440, 443-444, 467, 471, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 605-606, 608, 613-614, 619-620, 628, 641-642, 648-649, 678, 686, 695-696, 698, 700-701, 706-707, 710, 718-719, 729-730, 738, 741-742, 744, 752, 757, 762, 766, 772-774, 776, 779, 802, 811, 818, 820-821, 823, 827, 829, 834, 866, 869, 872, 874, 892, 905, 912, 914, 920, 925, 931, 939, 943, 947-948, 956-957, 959, 962, 967, 969, 976-977, 981, 983, 986, 989, 993, 995, 1002, 1015, 1022], plc=IGNORE] [2018-11-14 09:49:44,064][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:44,065][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:44,066][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,066][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,067][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,067][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,067][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:44,067][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0]] [2018-11-14 09:49:44,068][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:44,068][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 3 ms. [2018-11-14 09:49:44,069][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 4 ms. [2018-11-14 09:49:44,069][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:44,068][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity applying from full message performed in 2 ms. [2018-11-14 09:49:44,069][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:44,069][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], err=null] [2018-11-14 09:49:44,070][WARN ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[7, 18-19, 22, 24, 26, 29, 31-32, 39-40, 42, 56-57, 59, 61-63, 69-71, 78-80, 86, 88, 101, 107, 109, 114-115, 121, 130, 137, 141, 153, 155, 157, 161, 166, 172, 175, 177-179, 184, 187-188, 194-195, 198, 205, 209, 211, 213, 215, 220, 224, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 329-330, 337, 341, 343, 349, 352, 363, 373, 381-383, 392, 401-402, 407, 422, 426, 429-430, 432, 435, 440, 443-444, 467, 471, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 605-606, 608, 613-614, 619-620, 628, 641-642, 648-649, 678, 686, 695-696, 698, 700-701, 706-707, 710, 718-719, 729-730, 738, 741-742, 744, 752, 757, 762, 766, 772-774, 776, 779, 802, 811, 818, 820-821, 823, 827, 829, 834, 866, 869, 872, 874, 892, 905, 912, 914, 920, 925, 931, 939, 943, 947-948, 956-957, 959, 962, 967, 969, 976-977, 981, 983, 986, 989, 993, 995, 1002, 1015, 1022], plc=IGNORE] [2018-11-14 09:49:44,072][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,072][INFO ][sys-#98514%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], durationFromInit=21] [2018-11-14 09:49:44,069][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], err=null] [2018-11-14 09:49:44,069][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:44,076][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 8 ms. [2018-11-14 09:49:44,068][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:44,077][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,077][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,073][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=9, minorTopVer=0], force=false, evt=NODE_FAILED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:44,069][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 3 ms. [2018-11-14 09:49:44,078][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], err=null] [2018-11-14 09:49:44,078][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,080][WARN ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[7, 18-19, 22, 24, 26, 29, 31-32, 39-40, 42, 56-57, 59, 61-63, 69-71, 78-80, 86, 88, 101, 107, 109, 114-115, 121, 130, 137, 141, 153, 155, 157, 161, 166, 172, 175, 177-179, 184, 187-188, 194-195, 198, 205, 209, 211, 213, 215, 220, 224, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 329-330, 337, 341, 343, 349, 352, 363, 373, 381-383, 392, 401-402, 407, 422, 426, 429-430, 432, 435, 440, 443-444, 467, 471, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 605-606, 608, 613-614, 619-620, 628, 641-642, 648-649, 678, 686, 695-696, 698, 700-701, 706-707, 710, 718-719, 729-730, 738, 741-742, 744, 752, 757, 762, 766, 772-774, 776, 779, 802, 811, 818, 820-821, 823, 827, 829, 834, 866, 869, 872, 874, 892, 905, 912, 914, 920, 925, 931, 939, 943, 947-948, 956-957, 959, 962, 967, 969, 976-977, 981, 983, 986, 989, 993, 995, 1002, 1015, 1022], plc=IGNORE] [2018-11-14 09:49:44,079][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:44,079][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,079][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=9, minorTopVer=0], force=false, evt=NODE_FAILED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:44,082][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], err=null] [2018-11-14 09:49:44,083][WARN ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[7, 18-19, 22, 24, 26, 29, 31-32, 39-40, 42, 56-57, 59, 61-63, 69-71, 78-80, 86, 88, 101, 107, 109, 114-115, 121, 130, 137, 141, 153, 155, 157, 161, 166, 172, 175, 177-179, 184, 187-188, 194-195, 198, 205, 209, 211, 213, 215, 220, 224, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 329-330, 337, 341, 343, 349, 352, 363, 373, 381-383, 392, 401-402, 407, 422, 426, 429-430, 432, 435, 440, 443-444, 467, 471, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 605-606, 608, 613-614, 619-620, 628, 641-642, 648-649, 678, 686, 695-696, 698, 700-701, 706-707, 710, 718-719, 729-730, 738, 741-742, 744, 752, 757, 762, 766, 772-774, 776, 779, 802, 811, 818, 820-821, 823, 827, 829, 834, 866, 869, 872, 874, 892, 905, 912, 914, 920, 925, 931, 939, 943, 947-948, 956-957, 959, 962, 967, 969, 976-977, 981, 983, 986, 989, 993, 995, 1002, 1015, 1022], plc=IGNORE] [2018-11-14 09:49:44,085][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,085][INFO ][sys-#98516%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983362, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:44,081][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:44,086][INFO ][sys-#98599%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:44,080][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], err=null] [2018-11-14 09:49:44,087][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=9, minorTopVer=0], force=false, evt=NODE_FAILED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:44,086][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=9, minorTopVer=0], force=false, evt=NODE_FAILED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:44,083][INFO ][tcp-disco-msg-worker-#26434%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=0, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,088][WARN ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[7, 18-19, 22, 24, 26, 29, 31-32, 39-40, 42, 56-57, 59, 61-63, 69-71, 78-80, 86, 88, 101, 107, 109, 114-115, 121, 130, 137, 141, 153, 155, 157, 161, 166, 172, 175, 177-179, 184, 187-188, 194-195, 198, 205, 209, 211, 213, 215, 220, 224, 235-236, 238, 243, 250, 256, 260, 264, 270, 273, 277, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 329-330, 337, 341, 343, 349, 352, 363, 373, 381-383, 392, 401-402, 407, 422, 426, 429-430, 432, 435, 440, 443-444, 467, 471, 482-483, 506, 512, 515-516, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 575, 577-578, 580, 587-588, 599-600, 602, 605-606, 608, 613-614, 619-620, 628, 641-642, 648-649, 678, 686, 695-696, 698, 700-701, 706-707, 710, 718-719, 729-730, 738, 741-742, 744, 752, 757, 762, 766, 772-774, 776, 779, 802, 811, 818, 820-821, 823, 827, 829, 834, 866, 869, 872, 874, 892, 905, 912, 914, 920, 925, 931, 939, 943, 947-948, 956-957, 959, 962, 967, 969, 976-977, 981, 983, 986, 989, 993, 995, 1002, 1015, 1022], plc=IGNORE] [2018-11-14 09:49:44,089][INFO ][tcp-disco-srvr-#26444%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=56023] [2018-11-14 09:49:44,089][INFO ][tcp-disco-sock-reader-#26438%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:52586, rmtPort=52586 [2018-11-14 09:49:44,090][INFO ][tcp-disco-srvr-#26444%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=56023] [2018-11-14 09:49:44,090][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,091][INFO ][tcp-disco-sock-reader-#26446%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:56023, rmtPort=56023] [2018-11-14 09:49:44,091][INFO ][sys-#98508%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=ab7119de-eeba-4180-b532-10ed81e00004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=6, intOrder=6, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=9, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:44,092][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=9, minorTopVer=0], force=false, evt=NODE_FAILED, node=ab7119de-eeba-4180-b532-10ed81e00004] [2018-11-14 09:49:44,099][INFO ][tcp-disco-msg-worker-#26443%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,099][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=59382] [2018-11-14 09:49:44,100][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=59382] [2018-11-14 09:49:44,100][INFO ][tcp-disco-sock-reader-#26447%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:59382, rmtPort=59382] [2018-11-14 09:49:44,101][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,102][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=10, locNode=3dc77d76, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,102][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,102][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,103][INFO ][disco-event-worker-#98489%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=10, locNode=f3cd78d0, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,104][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,104][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,103][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,103][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,106][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,103][INFO ][disco-event-worker-#98475%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=10, locNode=2f28edb7, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,105][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=10, locNode=4ce119dc, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,104][INFO ][disco-notifier-worker-#98640%tcp.TcpDiscoveryMultiThreadedTest4%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,104][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,108][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=10, locNode=77abe0df, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,109][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,107][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,107][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,110][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,109][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=127858MB, available=96649MB] [2018-11-14 09:49:44,109][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,110][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,112][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]]] [2018-11-14 09:49:44,112][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,113][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,113][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983372, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]]] [2018-11-14 09:49:44,115][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,115][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:44,113][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]]] [2018-11-14 09:49:44,116][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,113][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]]] [2018-11-14 09:49:44,113][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]]] [2018-11-14 09:49:44,117][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,118][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,119][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=false] [2018-11-14 09:49:44,117][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,116][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][msg] Components activation performed in 1 ms. [2018-11-14 09:49:44,116][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,115][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,120][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=false] [2018-11-14 09:49:44,119][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=false] [2018-11-14 09:49:44,119][INFO ][disco-notifier-worker-#98640%tcp.TcpDiscoveryMultiThreadedTest4%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:44,118][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,120][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=false] [2018-11-14 09:49:44,121][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0], node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, allReceived=false] [2018-11-14 09:49:44,122][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=false] [2018-11-14 09:49:44,122][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0], node=f3cd78d0-c516-4ff3-8c99-885d58e00001, allReceived=false] [2018-11-14 09:49:44,123][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:44,123][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=true] [2018-11-14 09:49:44,126][INFO ][sys-#98648%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:44,126][INFO ][sys-#98649%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:44,127][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Starting caches on local join performed in 8 ms. [2018-11-14 09:49:44,127][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,128][INFO ][grid-nio-worker-tcp-comm-2-#98272%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36544] [2018-11-14 09:49:44,129][INFO ][grid-nio-worker-tcp-comm-0-#98628%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36544, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:44,129][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:44,129][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], crd=false] [2018-11-14 09:49:44,129][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0], node=5017e3fa-a0f1-4c55-ae20-110197800004, allReceived=true] [2018-11-14 09:49:44,130][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,130][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:44,130][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,131][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,131][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 1 ms. [2018-11-14 09:49:44,132][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:44,132][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:44,133][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms. [2018-11-14 09:49:44,134][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 1 ms. [2018-11-14 09:49:44,134][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 1 ms. [2018-11-14 09:49:44,134][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,134][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,135][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:44,137][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,137][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,138][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,139][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:44,139][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,139][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:44,140][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,140][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,141][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,141][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,141][INFO ][exchange-worker-#98476%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,142][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,143][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:44,135][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,142][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 3 ms. [2018-11-14 09:49:44,141][INFO ][sys-#98507%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=31] [2018-11-14 09:49:44,137][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,144][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,145][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,146][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,147][INFO ][sys-#98491%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:44,144][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,147][INFO ][exchange-worker-#98490%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,146][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,148][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,149][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 4 ms. [2018-11-14 09:49:44,149][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:44,150][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,149][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:44,151][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,151][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,152][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]] [2018-11-14 09:49:44,152][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,152][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=41] [2018-11-14 09:49:44,155][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][CacheAffinitySharedManager] Affinity initialization on local join performed in 3 ms. [2018-11-14 09:49:44,155][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 3 ms. [2018-11-14 09:49:44,157][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,157][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], err=null] [2018-11-14 09:49:44,158][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,158][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,158][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,159][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=5017e3fa-a0f1-4c55-ae20-110197800004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984155, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], durationFromInit=51] [2018-11-14 09:49:44,160][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=10, minorTopVer=0], force=false, evt=NODE_JOINED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,161][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,161][INFO ][sys-#98645%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:44,162][INFO ][utility-#98654%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,163][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:44,163][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:44,164][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=77abe0df-a676-4745-b311-5aa4bb000002, partitionsCount=43, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,164][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=41, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,165][INFO ][grid-nio-worker-tcp-comm-1-#98569%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45183, rmtAddr=/127.0.0.1:46902] [2018-11-14 09:49:44,165][INFO ][sys-#98286%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,165][INFO ][sys-#98653%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[19, 32, 69, 79, 121, 157, 175, 188, 198, 211, 243, 260, 273, 281, 285, 341, 343, 349, 426, 429, 440, 482, 530, 556, 575, 613, 641, 678, 698, 700-701, 730, 762, 774, 818, 821, 948, 956, 976-977, 989], histPartitions=[]] [2018-11-14 09:49:44,165][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:44,166][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Data Regions Configured: [2018-11-14 09:49:44,166][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:44,167][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest4 >>> Local node [ID=5017E3FA-A0F1-4C55-AE20-110197800004, order=10, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10805 TCP:45185 TCP:47505 [2018-11-14 09:49:44,167][INFO ][node-restart-4][GridDiscoveryManager] Topology snapshot [ver=10, locNode=5017e3fa, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,168][INFO ][node-restart-4][root] Node started with the following configuration [id=5017e3fa-a0f1-4c55-ae20-110197800004, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@423722fd], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:44,165][INFO ][grid-nio-worker-tcp-comm-1-#98629%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:46902, rmtAddr=/127.0.0.1:45183] [2018-11-14 09:49:44,169][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, partitionsCount=29, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,169][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,170][INFO ][grid-nio-worker-tcp-comm-1-#98406%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45182, rmtAddr=/127.0.0.1:59798] [2018-11-14 09:49:44,170][INFO ][grid-nio-worker-tcp-comm-2-#98630%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:59798, rmtAddr=/127.0.0.1:45182] [2018-11-14 09:49:44,170][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=f3cd78d0-c516-4ff3-8c99-885d58e00001, partitionsCount=38, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,169][INFO ][sys-#98646%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topic=0, fullPartitions=[130, 153, 155, 166, 177, 194, 215, 235, 290, 307, 314-315, 330, 363, 373, 382, 432, 443, 483, 515, 535, 559, 565, 600, 686, 696, 719, 742, 752, 757, 773, 802, 811, 827, 834, 914, 939, 959, 967, 969, 993, 1015, 1022], histPartitions=[]] [2018-11-14 09:49:44,171][INFO ][grid-nio-worker-tcp-comm-1-#98401%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45181, rmtAddr=/127.0.0.1:34750] [2018-11-14 09:49:44,171][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, partitionsCount=26, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,171][INFO ][sys-#98481%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,174][INFO ][grid-nio-worker-tcp-comm-1-#98425%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45184, rmtAddr=/127.0.0.1:58682] [2018-11-14 09:49:44,173][INFO ][sys-#98652%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=1/5, time=10 ms] [2018-11-14 09:49:44,173][INFO ][sys-#98648%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, topic=0, fullPartitions=[18, 26, 29, 40, 42, 56, 80, 86, 101, 107, 141, 209, 310, 383, 392, 430, 516, 526, 534, 550, 578, 580, 706, 710, 872, 905, 925, 943, 957], histPartitions=[]] [2018-11-14 09:49:44,172][INFO ][grid-nio-worker-tcp-comm-3-#98631%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:34750, rmtAddr=/127.0.0.1:45181] [2018-11-14 09:49:44,174][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=2/5, time=10 ms] [2018-11-14 09:49:44,174][INFO ][grid-nio-worker-tcp-comm-0-#98628%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:58682, rmtAddr=/127.0.0.1:45184] [2018-11-14 09:49:44,176][INFO ][sys-#98649%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=f3cd78d0-c516-4ff3-8c99-885d58e00001, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=3/5, time=20 ms] [2018-11-14 09:49:44,175][INFO ][sys-#98495%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,176][INFO ][sys-#98653%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=4/5, time=20 ms] [2018-11-14 09:49:44,176][INFO ][sys-#98524%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=5017e3fa-a0f1-4c55-ae20-110197800004, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], topic=0] [2018-11-14 09:49:44,177][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], progress=5/5, time=20 ms] [2018-11-14 09:49:44,177][INFO ][sys-#98650%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=f3cd78d0-c516-4ff3-8c99-885d58e00001, topic=0, fullPartitions=[22, 57, 62, 71, 88, 178, 184, 238, 250, 264, 270, 292, 300, 337, 352, 401-402, 422, 444, 512, 531, 554, 566, 587-588, 599, 602, 620, 744, 772, 776, 823, 869, 892, 912, 920, 986, 995], histPartitions=[]] [2018-11-14 09:49:44,177][INFO ][sys-#98645%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, topic=0, fullPartitions=[39, 59, 61, 63, 109, 114, 137, 161, 179, 187, 213, 236, 256, 277, 293, 297, 407, 467, 506, 577, 642, 695, 718, 741, 829, 931], histPartitions=[]] [2018-11-14 09:49:44,178][INFO ][utility-#98655%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0], rebalanceId=1, routines=5] [2018-11-14 09:49:44,311][WARN ][node-restart-3][TcpDiscoverySpi] Simulating node failure: 2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:44,312][ERROR][tcp-disco-srvr-#26399%tcp.TcpDiscoveryMultiThreadedTest3%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,314][INFO ][tcp-disco-sock-reader-#26412%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:49057, rmtPort=49057 [2018-11-14 09:49:44,314][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryDiscardMessage [msgId=c6c03a11761-f3cd78d0-c516-4ff3-8c99-885d58e00001, customMsgDiscard=true, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=d5d03a11761-3dc77d76-35d6-4baf-b057-c1ecfcf00000, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryDiscardMessage [msgId=c6c03a11761-f3cd78d0-c516-4ff3-8c99-885d58e00001, customMsgDiscard=true, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=d5d03a11761-3dc77d76-35d6-4baf-b057-c1ecfcf00000, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, order=2, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:44,315][WARN ][node-restart-1][TcpDiscoverySpi] Simulating node failure: f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:44,317][ERROR][tcp-disco-srvr-#26401%tcp.TcpDiscoveryMultiThreadedTest1%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,315][ERROR][tcp-disco-msg-worker-#26398%tcp.TcpDiscoveryMultiThreadedTest3%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,318][INFO ][tcp-disco-sock-reader-#26431%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:41008, rmtPort=41008 [2018-11-14 09:49:44,316][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,321][ERROR][tcp-disco-msg-worker-#26397%tcp.TcpDiscoveryMultiThreadedTest1%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,322][INFO ][node-restart-3][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:44,322][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryDiscardMessage [msgId=c6c03a11761-f3cd78d0-c516-4ff3-8c99-885d58e00001, customMsgDiscard=true, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=d5d03a11761-3dc77d76-35d6-4baf-b057-c1ecfcf00000, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryDiscardMessage [msgId=c6c03a11761-f3cd78d0-c516-4ff3-8c99-885d58e00001, customMsgDiscard=true, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=d5d03a11761-3dc77d76-35d6-4baf-b057-c1ecfcf00000, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, order=4, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:44,324][INFO ][node-restart-1][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:44,323][INFO ][node-restart-3][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:44,325][INFO ][node-restart-1][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:44,324][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,326][INFO ][node-restart-1][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983412, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=e77f2a11761-f3cd78d0-c516-4ff3-8c99-885d58e00001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:44,326][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=43060] [2018-11-14 09:49:44,326][INFO ][node-restart-3][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983412, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=e87f2a11761-2f28edb7-bf70-4ca2-9cb7-d18c1f500003, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:44,327][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=43060] [2018-11-14 09:49:44,327][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45181, failureDetectionTimeout=10000] [2018-11-14 09:49:44,328][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:44,328][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001, res=false, time=2ms] [2018-11-14 09:49:44,329][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:44,329][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45182, failureDetectionTimeout=10000] [2018-11-14 09:49:44,329][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45182]] [2018-11-14 09:49:44,330][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:44,330][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, res=false, time=4ms] [2018-11-14 09:49:44,330][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): 2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:44,328][INFO ][tcp-disco-sock-reader-#26449%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:43060, rmtPort=43060] [2018-11-14 09:49:44,328][WARN ][tcp-comm-worker-#26441%tcp.TcpDiscoveryMultiThreadedTest4%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45181]] [2018-11-14 09:49:44,330][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest3 >>> Grid uptime: 00:00:00.832 [2018-11-14 09:49:44,328][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest1 >>> Grid uptime: 00:00:00.832 [2018-11-14 09:49:44,332][INFO ][tcp-disco-sock-reader-#26449%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Connection check done: [liveAddr=null, previousNode=TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47502], connectingNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,333][INFO ][tcp-disco-sock-reader-#26449%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=1542188984306, now=1542188984326, connCheckInterval=500] [2018-11-14 09:49:44,331][INFO ][tcp-comm-worker-#26441%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Pinging node: f3cd78d0-c516-4ff3-8c99-885d58e00001 [2018-11-14 09:49:44,334][INFO ][tcp-comm-worker-#26441%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished node ping [nodeId=f3cd78d0-c516-4ff3-8c99-885d58e00001, res=false, time=8ms] [2018-11-14 09:49:44,335][INFO ][tcp-comm-worker-#26441%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Pinging node: 2f28edb7-bf70-4ca2-9cb7-d18c1f500003 [2018-11-14 09:49:44,335][INFO ][tcp-comm-worker-#26441%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished node ping [nodeId=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, res=false, time=9ms] [2018-11-14 09:49:44,341][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,341][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Config URL: n/a [2018-11-14 09:49:44,341][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest1, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=ae293efb-8b41-4b86-b32b-747070800001, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@257ce171, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@24fe791b[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@1cc663b8, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@3aed72b3, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@190bb636, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,347][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Daemon mode: off [2018-11-14 09:49:44,347][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,347][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] OS user: teamcity [2018-11-14 09:49:44,342][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,348][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Config URL: n/a [2018-11-14 09:49:44,347][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] PID: 18275 [2018-11-14 09:49:44,348][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,348][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,348][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest3, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@c4c12fe, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@31b198d7[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4159bf6c, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@73073b93, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@77accd41, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Daemon mode: off [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] OS user: teamcity [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] PID: 18275 [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,355][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,356][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM total memory: 1.8GB [2018-11-14 09:49:44,356][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,356][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,356][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,356][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,359][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,360][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,360][WARN ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,349][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM total memory: 1.8GB [2018-11-14 09:49:44,360][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,360][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,361][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,361][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,364][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,364][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,364][WARN ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,365][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,365][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,366][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=11, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,366][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,367][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,367][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,367][INFO ][node-restart-1][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,368][INFO ][node-restart-1][IgnitePluginProcessor] [2018-11-14 09:49:44,369][INFO ][node-restart-1][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,369][INFO ][node-restart-3][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,369][INFO ][node-restart-3][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,369][INFO ][node-restart-3][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,369][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:44,369][INFO ][node-restart-3][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,370][INFO ][node-restart-3][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,370][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:44,370][INFO ][node-restart-3][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,370][INFO ][node-restart-3][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,370][INFO ][node-restart-3][IgnitePluginProcessor] [2018-11-14 09:49:44,370][INFO ][node-restart-3][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,371][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,371][INFO ][node-restart-3][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45182, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,372][WARN ][node-restart-3][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,371][INFO ][node-restart-1][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45181, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,371][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=12, locNode=3dc77d76, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,373][WARN ][node-restart-1][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,377][WARN ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,377][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=11, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,378][WARN ][node-restart-1][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,378][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,380][WARN ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,380][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=11, locNode=5017e3fa, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,381][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,382][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,377][WARN ][node-restart-3][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,380][INFO ][node-restart-1][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,378][WARN ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,383][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=11, locNode=77abe0df, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,383][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,384][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,378][WARN ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,378][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,386][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,386][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=12, locNode=4ce119dc, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,384][WARN ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,383][WARN ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=f3cd78d0-c516-4ff3-8c99-885d58e00001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=4, intOrder=4, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,382][INFO ][node-restart-1][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,382][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,389][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0]]] [2018-11-14 09:49:44,390][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,390][INFO ][node-restart-3][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,390][INFO ][node-restart-3][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,391][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0]]] [2018-11-14 09:49:44,391][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,392][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,392][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], crd=false] [2018-11-14 09:49:44,398][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0]]] [2018-11-14 09:49:44,399][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,399][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,400][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], crd=false] [2018-11-14 09:49:44,388][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=12, locNode=77abe0df, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,388][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=12, locNode=5017e3fa, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,392][INFO ][node-restart-1][ClientListenerProcessor] Client connector processor has started on TCP port 10803 [2018-11-14 09:49:44,392][INFO ][node-restart-3][ClientListenerProcessor] Client connector processor has started on TCP port 10801 [2018-11-14 09:49:44,392][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0]]] [2018-11-14 09:49:44,403][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,403][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=11, minorTopVer=0], node=5017e3fa-a0f1-4c55-ae20-110197800004, allReceived=false] [2018-11-14 09:49:44,403][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=11, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=false] [2018-11-14 09:49:44,404][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=11, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=true] [2018-11-14 09:49:44,390][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,404][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], crd=false] [2018-11-14 09:49:44,404][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=11, minorTopVer=0]] [2018-11-14 09:49:44,405][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=12, minorTopVer=0], evt=NODE_FAILED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:44,405][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:44,405][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,406][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,406][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,407][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:44,408][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:44,408][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,408][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,408][INFO ][node-restart-1][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,408][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:44,409][WARN ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0, 6, 12, 14, 17-18, 22-23, 26-27, 29, 36-37, 40, 42, 44-46, 48-49, 51, 56-58, 60, 62, 65-67, 71, 73-77, 80, 83-84, 86, 88, 90, 92, 94-96, 98-99, 101, 103-105, 107, 112-113, 117-118, 120, 123, 125, 131, 133, 136, 138-141, 143-147, 156, 162-163, 167, 169-170, 173-174, 178, 180, 183-184, 190, 192-193, 200-201, 203-204, 208-209, 212, 219, 221, 223, 227, 230-233, 238, 240, 244, 246, 249-250, 252-255, 257-259, 262, 264-265, 270, 272, 276, 278, 287-288, 292, 294, 296, 298-300, 303, 308-310, 318-323, 325-328, 332-333, 337-340, 342, 344, 346-348, 350-352, 354-357, 361-362, 369, 374, 379, 383, 386, 388-389, 392, 400-402, 406, 408-409, 413, 419-420, 422-425, 430-431, 444, 447, 453-454, 458-459, 463-464, 466, 469, 473-474, 477, 480, 485, 487-488, 490-492, 496-498, 502-503, 509, 512, 516-517, 520-521, 523, 526-527, 531-534, 536, 538-539, 541-543, 545, 547-550, 552, 554-555, 561, 563, 566, 569-570, 576, 578-581, 585, 587-588, 596-597, 599, 601-602, 604, 611-612, 617-618, 620-622, 624, 627, 630, 637, 639, 643-644, 650-651, 655, 657-658, 660-663, 665, 668, 671-677, 679-680, 682, 687-689, 691, 693, 699, 702, 704-706, 710, 712-713, 716-717, 720, 722, 724-725, 734, 736-737, 743-751, 754, 759, 767-769, 771-772, 775-778, 780, 788, 791, 793, 795, 805-808, 812, 814, 817, 819, 822-823, 828, 830, 832, 843, 846-849, 853-854, 856, 859-860, 862, 864, 869, 872-873, 875-876, 880, 885, 887, 889-892, 895-896, 899, 903, 905, 908, 911-912, 915-916, 919-920, 922, 924-925, 927, 930, 933, 936, 940-941, 943, 946, 949, 953-955, 957, 964, 966, 970, 972, 975, 979, 982, 984, 986-987, 990-991, 994-996, 1003-1010, 1012-1013, 1017], plc=IGNORE] [2018-11-14 09:49:44,411][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,409][INFO ][node-restart-3][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,413][INFO ][node-restart-3][TcpDiscoverySpi] Successfully bound to TCP port [port=47503, localHost=/127.0.0.1, locNodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,413][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:44,413][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:44,413][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:44,414][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], err=null] [2018-11-14 09:49:44,414][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,414][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,415][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,415][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=12, minorTopVer=0], evt=NODE_FAILED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:44,416][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:44,417][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=46096] [2018-11-14 09:49:44,417][INFO ][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=46096] [2018-11-14 09:49:44,409][INFO ][node-restart-1][TcpDiscoverySpi] Successfully bound to TCP port [port=47502, localHost=/127.0.0.1, locNodeId=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:44,418][INFO ][tcp-disco-sock-reader-#26457%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:46096, rmtPort=46096] [2018-11-14 09:49:44,418][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 3 ms. [2018-11-14 09:49:44,417][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983342, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], durationFromInit=54] [2018-11-14 09:49:44,415][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,419][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,420][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=12, minorTopVer=0], evt=NODE_FAILED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:44,414][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,421][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0]] [2018-11-14 09:49:44,419][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], crd=true] [2018-11-14 09:49:44,418][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:44,421][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], err=null] [2018-11-14 09:49:44,422][WARN ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0, 6, 12, 14, 17-18, 22-23, 26-27, 29, 36-37, 40, 42, 44-46, 48-49, 51, 56-58, 60, 62, 65-67, 71, 73-77, 80, 83-84, 86, 88, 90, 92, 94-96, 98-99, 101, 103-105, 107, 112-113, 117-118, 120, 123, 125, 131, 133, 136, 138-141, 143-147, 156, 162-163, 167, 169-170, 173-174, 178, 180, 183-184, 190, 192-193, 200-201, 203-204, 208-209, 212, 219, 221, 223, 227, 230-233, 238, 240, 244, 246, 249-250, 252-255, 257-259, 262, 264-265, 270, 272, 276, 278, 287-288, 292, 294, 296, 298-300, 303, 308-310, 318-323, 325-328, 332-333, 337-340, 342, 344, 346-348, 350-352, 354-357, 361-362, 369, 374, 379, 383, 386, 388-389, 392, 400-402, 406, 408-409, 413, 419-420, 422-425, 430-431, 444, 447, 453-454, 458-459, 463-464, 466, 469, 473-474, 477, 480, 485, 487-488, 490-492, 496-498, 502-503, 509, 512, 516-517, 520-521, 523, 526-527, 531-534, 536, 538-539, 541-543, 545, 547-550, 552, 554-555, 561, 563, 566, 569-570, 576, 578-581, 585, 587-588, 596-597, 599, 601-602, 604, 611-612, 617-618, 620-622, 624, 627, 630, 637, 639, 643-644, 650-651, 655, 657-658, 660-663, 665, 668, 671-677, 679-680, 682, 687-689, 691, 693, 699, 702, 704-706, 710, 712-713, 716-717, 720, 722, 724-725, 734, 736-737, 743-751, 754, 759, 767-769, 771-772, 775-778, 780, 788, 791, 793, 795, 805-808, 812, 814, 817, 819, 822-823, 828, 830, 832, 843, 846-849, 853-854, 856, 859-860, 862, 864, 869, 872-873, 875-876, 880, 885, 887, 889-892, 895-896, 899, 903, 905, 908, 911-912, 915-916, 919-920, 922, 924-925, 927, 930, 933, 936, 940-941, 943, 946, 949, 953-955, 957, 964, 966, 970, 972, 975, 979, 982, 984, 986-987, 990-991, 994-996, 1003-1010, 1012-1013, 1017], plc=IGNORE] [2018-11-14 09:49:44,421][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:44,421][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=11, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=12, minorTopVer=0], evt=NODE_FAILED, evtNode=f3cd78d0-c516-4ff3-8c99-885d58e00001, evtNodeClient=false] [2018-11-14 09:49:44,424][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,425][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:44,426][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 5 ms. [2018-11-14 09:49:44,424][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=12, minorTopVer=0], force=false, evt=NODE_FAILED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:44,424][INFO ][sys-#98649%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,424][INFO ][sys-#98526%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,424][INFO ][sys-#98598%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,423][INFO ][tcp-disco-sock-reader-#26457%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:46096, rmtPort=46096 [2018-11-14 09:49:44,422][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=57722] [2018-11-14 09:49:44,433][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=12, minorTopVer=0], force=false, evt=NODE_FAILED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:44,432][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], durationFromInit=52] [2018-11-14 09:49:44,434][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=57722] [2018-11-14 09:49:44,435][INFO ][tcp-disco-sock-reader-#26459%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:57722, rmtPort=57722] [2018-11-14 09:49:44,431][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 5 ms. [2018-11-14 09:49:44,435][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], err=null] [2018-11-14 09:49:44,436][WARN ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0, 6, 12, 14, 17-18, 22-23, 26-27, 29, 36-37, 40, 42, 44-46, 48-49, 51, 56-58, 60, 62, 65-67, 71, 73-77, 80, 83-84, 86, 88, 90, 92, 94-96, 98-99, 101, 103-105, 107, 112-113, 117-118, 120, 123, 125, 131, 133, 136, 138-141, 143-147, 156, 162-163, 167, 169-170, 173-174, 178, 180, 183-184, 190, 192-193, 200-201, 203-204, 208-209, 212, 219, 221, 223, 227, 230-233, 238, 240, 244, 246, 249-250, 252-255, 257-259, 262, 264-265, 270, 272, 276, 278, 287-288, 292, 294, 296, 298-300, 303, 308-310, 318-323, 325-328, 332-333, 337-340, 342, 344, 346-348, 350-352, 354-357, 361-362, 369, 374, 379, 383, 386, 388-389, 392, 400-402, 406, 408-409, 413, 419-420, 422-425, 430-431, 444, 447, 453-454, 458-459, 463-464, 466, 469, 473-474, 477, 480, 485, 487-488, 490-492, 496-498, 502-503, 509, 512, 516-517, 520-521, 523, 526-527, 531-534, 536, 538-539, 541-543, 545, 547-550, 552, 554-555, 561, 563, 566, 569-570, 576, 578-581, 585, 587-588, 596-597, 599, 601-602, 604, 611-612, 617-618, 620-622, 624, 627, 630, 637, 639, 643-644, 650-651, 655, 657-658, 660-663, 665, 668, 671-677, 679-680, 682, 687-689, 691, 693, 699, 702, 704-706, 710, 712-713, 716-717, 720, 722, 724-725, 734, 736-737, 743-751, 754, 759, 767-769, 771-772, 775-778, 780, 788, 791, 793, 795, 805-808, 812, 814, 817, 819, 822-823, 828, 830, 832, 843, 846-849, 853-854, 856, 859-860, 862, 864, 869, 872-873, 875-876, 880, 885, 887, 889-892, 895-896, 899, 903, 905, 908, 911-912, 915-916, 919-920, 922, 924-925, 927, 930, 933, 936, 940-941, 943, 946, 949, 953-955, 957, 964, 966, 970, 972, 975, 979, 982, 984, 986-987, 990-991, 994-996, 1003-1010, 1012-1013, 1017], plc=IGNORE] [2018-11-14 09:49:44,438][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,425][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 6 ms. [2018-11-14 09:49:44,440][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,440][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], err=null] [2018-11-14 09:49:44,440][WARN ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0, 6, 12, 14, 17-18, 22-23, 26-27, 29, 36-37, 40, 42, 44-46, 48-49, 51, 56-58, 60, 62, 65-67, 71, 73-77, 80, 83-84, 86, 88, 90, 92, 94-96, 98-99, 101, 103-105, 107, 112-113, 117-118, 120, 123, 125, 131, 133, 136, 138-141, 143-147, 156, 162-163, 167, 169-170, 173-174, 178, 180, 183-184, 190, 192-193, 200-201, 203-204, 208-209, 212, 219, 221, 223, 227, 230-233, 238, 240, 244, 246, 249-250, 252-255, 257-259, 262, 264-265, 270, 272, 276, 278, 287-288, 292, 294, 296, 298-300, 303, 308-310, 318-323, 325-328, 332-333, 337-340, 342, 344, 346-348, 350-352, 354-357, 361-362, 369, 374, 379, 383, 386, 388-389, 392, 400-402, 406, 408-409, 413, 419-420, 422-425, 430-431, 444, 447, 453-454, 458-459, 463-464, 466, 469, 473-474, 477, 480, 485, 487-488, 490-492, 496-498, 502-503, 509, 512, 516-517, 520-521, 523, 526-527, 531-534, 536, 538-539, 541-543, 545, 547-550, 552, 554-555, 561, 563, 566, 569-570, 576, 578-581, 585, 587-588, 596-597, 599, 601-602, 604, 611-612, 617-618, 620-622, 624, 627, 630, 637, 639, 643-644, 650-651, 655, 657-658, 660-663, 665, 668, 671-677, 679-680, 682, 687-689, 691, 693, 699, 702, 704-706, 710, 712-713, 716-717, 720, 722, 724-725, 734, 736-737, 743-751, 754, 759, 767-769, 771-772, 775-778, 780, 788, 791, 793, 795, 805-808, 812, 814, 817, 819, 822-823, 828, 830, 832, 843, 846-849, 853-854, 856, 859-860, 862, 864, 869, 872-873, 875-876, 880, 885, 887, 889-892, 895-896, 899, 903, 905, 908, 911-912, 915-916, 919-920, 922, 924-925, 927, 930, 933, 936, 940-941, 943, 946, 949, 953-955, 957, 964, 966, 970, 972, 975, 979, 982, 984, 986-987, 990-991, 994-996, 1003-1010, 1012-1013, 1017], plc=IGNORE] [2018-11-14 09:49:44,442][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,446][INFO ][sys-#98651%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=5017e3fa-a0f1-4c55-ae20-110197800004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], durationFromInit=62] [2018-11-14 09:49:44,437][INFO ][tcp-disco-sock-reader-#26459%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:57722, rmtPort=57722 [2018-11-14 09:49:44,437][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=57714] [2018-11-14 09:49:44,448][INFO ][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=57714] [2018-11-14 09:49:44,447][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=12, minorTopVer=0], force=false, evt=NODE_FAILED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:44,445][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=12, minorTopVer=0], force=false, evt=NODE_FAILED, node=2f28edb7-bf70-4ca2-9cb7-d18c1f500003] [2018-11-14 09:49:44,443][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=11, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=2f28edb7-bf70-4ca2-9cb7-d18c1f500003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=2, intOrder=2, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=12, minorTopVer=0], durationFromInit=72] [2018-11-14 09:49:44,441][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,449][INFO ][tcp-disco-sock-reader-#26460%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:57714, rmtPort=57714] [2018-11-14 09:49:44,452][INFO ][tcp-disco-sock-reader-#26460%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:57714, rmtPort=57714 [2018-11-14 09:49:44,455][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:44,456][INFO ][tcp-disco-msg-worker-#26443%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=0, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,456][INFO ][tcp-disco-sock-reader-#26447%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:59382, rmtPort=59382 [2018-11-14 09:49:44,457][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=39424] [2018-11-14 09:49:44,457][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=39424] [2018-11-14 09:49:44,458][INFO ][tcp-disco-sock-reader-#26461%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39424, rmtPort=39424] [2018-11-14 09:49:44,465][INFO ][tcp-disco-msg-worker-#26455%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,466][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=57154] [2018-11-14 09:49:44,466][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=57154] [2018-11-14 09:49:44,467][INFO ][tcp-disco-sock-reader-#26462%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:57154, rmtPort=57154] [2018-11-14 09:49:44,468][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,468][INFO ][tcp-disco-msg-worker-#26455%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=0, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,469][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,469][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=13, locNode=5017e3fa, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,470][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=10, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,468][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=13, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,470][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,470][INFO ][tcp-disco-sock-reader-#26462%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:57154, rmtPort=57154 [2018-11-14 09:49:44,469][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=56340] [2018-11-14 09:49:44,469][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,472][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=13, locNode=77abe0df, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,469][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,472][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,472][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=56340] [2018-11-14 09:49:44,471][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_JOINED, evtNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,473][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,473][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=13, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,474][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,475][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_JOINED, evtNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,476][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,476][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]]] [2018-11-14 09:49:44,477][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,477][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983392, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]]] [2018-11-14 09:49:44,476][INFO ][tcp-disco-sock-reader-#26463%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:56340, rmtPort=56340] [2018-11-14 09:49:44,478][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,479][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,479][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], crd=false] [2018-11-14 09:49:44,477][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188983707, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]]] [2018-11-14 09:49:44,480][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,481][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,481][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], crd=false] [2018-11-14 09:49:44,477][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984094, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]]] [2018-11-14 09:49:44,482][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,483][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:44,483][INFO ][exchange-worker-#98644%tcp.TcpDiscoveryMultiThreadedTest4%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], crd=false] [2018-11-14 09:49:44,477][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], crd=true] [2018-11-14 09:49:44,480][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=false] [2018-11-14 09:49:44,484][INFO ][sys-#98286%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=5017e3fa-a0f1-4c55-ae20-110197800004, allReceived=false] [2018-11-14 09:49:44,484][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=false] [2018-11-14 09:49:44,488][INFO ][disco-notifier-worker-#98730%tcp.TcpDiscoveryMultiThreadedTest3%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,489][INFO ][tcp-disco-msg-worker-#26454%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,490][WARN ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=106537MB, available=96649MB] [2018-11-14 09:49:44,490][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=48800] [2018-11-14 09:49:44,490][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=48800] [2018-11-14 09:49:44,491][INFO ][tcp-disco-sock-reader-#26465%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:48800, rmtPort=48800] [2018-11-14 09:49:44,492][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,492][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,493][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=14, locNode=3dc77d76, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,493][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,494][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=14, locNode=4ce119dc, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,492][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:44,494][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][msg] Components activation performed in 2 ms. [2018-11-14 09:49:44,495][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,495][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=14, locNode=77abe0df, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,497][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,497][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,498][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=14, locNode=99dc07b8, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,497][INFO ][disco-event-worker-#98643%tcp.TcpDiscoveryMultiThreadedTest4%][GridDiscoveryManager] Topology snapshot [ver=14, locNode=5017e3fa, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,499][INFO ][disco-notifier-worker-#98729%tcp.TcpDiscoveryMultiThreadedTest1%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,499][INFO ][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:44,499][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:44,500][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Starting caches on local join performed in 6 ms. [2018-11-14 09:49:44,500][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]] [2018-11-14 09:49:44,594][INFO ][tcp-disco-sock-reader-#26437%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39028, rmtPort=39028 [2018-11-14 09:49:44,594][INFO ][tcp-disco-sock-reader-#26424%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:36815, rmtPort=36815 [2018-11-14 09:49:44,595][INFO ][grid-nio-worker-tcp-comm-3-#98273%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36568] [2018-11-14 09:49:44,595][INFO ][grid-nio-worker-tcp-comm-0-#98707%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36568, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:44,596][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:44,596][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], crd=false] [2018-11-14 09:49:44,596][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, allReceived=true] [2018-11-14 09:49:44,596][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=13, minorTopVer=0]] [2018-11-14 09:49:44,597][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=ae293efb-8b41-4b86-b32b-747070800001, evtNodeClient=false] [2018-11-14 09:49:44,597][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, wait for message [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:44,597][INFO ][sys-#98285%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:44,600][WARN ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=127837MB, available=96649MB] [2018-11-14 09:49:44,602][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=ae293efb-8b41-4b86-b32b-747070800001, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,603][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:44,603][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][msg] Components activation performed in 0 ms. [2018-11-14 09:49:44,609][WARN ][node-restart-4][TcpDiscoverySpi] Simulating node failure: 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,609][ERROR][tcp-disco-srvr-#26444%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,611][INFO ][tcp-disco-sock-reader-#26446%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:56023, rmtPort=56023 [2018-11-14 09:49:44,612][ERROR][tcp-disco-msg-worker-#26443%tcp.TcpDiscoveryMultiThreadedTest4%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,618][WARN ][tcp-disco-msg-worker-#26434%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, id=dd013a11761-77abe0df-a676-4745-b311-5aa4bb000002, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=14, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=null, super=TcpDiscoveryAbstractMessage [sndNodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, id=dd013a11761-77abe0df-a676-4745-b311-5aa4bb000002, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=14, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, order=10, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:44,619][INFO ][tcp-disco-msg-worker-#26434%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,619][INFO ][node-restart-4][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], resVer=null, err=class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is stopping: tcp.TcpDiscoveryMultiThreadedTest4] [2018-11-14 09:49:44,620][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36339] [2018-11-14 09:49:44,620][INFO ][node-restart-4][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=5017e3fa-a0f1-4c55-ae20-110197800004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=null, durationFromInit=1542188984614] [2018-11-14 09:49:44,621][INFO ][node-restart-4][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=null, err=class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is stopping: tcp.TcpDiscoveryMultiThreadedTest4] [2018-11-14 09:49:44,621][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36339] [2018-11-14 09:49:44,621][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:44,622][INFO ][sys-#98744%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:44,622][INFO ][tcp-disco-sock-reader-#26467%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:36339, rmtPort=36339] [2018-11-14 09:49:44,622][INFO ][node-restart-4][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=5017e3fa-a0f1-4c55-ae20-110197800004, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=null, durationFromInit=153] [2018-11-14 09:49:44,623][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Starting caches on local join performed in 20 ms. [2018-11-14 09:49:44,624][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,624][INFO ][node-restart-4][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:44,625][INFO ][tcp-disco-sock-reader-#26467%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Connection check done: [liveAddr=null, previousNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47505], connectingNodeId=77abe0df-a676-4745-b311-5aa4bb000002] [2018-11-14 09:49:44,625][INFO ][grid-nio-worker-tcp-comm-0-#98270%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36572] [2018-11-14 09:49:44,625][INFO ][node-restart-4][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:44,625][INFO ][tcp-disco-sock-reader-#26467%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=1542188984604, now=1542188984614, connCheckInterval=500] [2018-11-14 09:49:44,626][INFO ][grid-nio-worker-tcp-comm-0-#98703%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36572, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:44,626][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:44,627][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], crd=false] [2018-11-14 09:49:44,626][INFO ][node-restart-4][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188984114, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=37603a11761-5017e3fa-a0f1-4c55-ae20-110197800004, userVer=0, loc=true, sampleClsName=org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:44,627][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, received message [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001, msgVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], process=true, awaited=1] [2018-11-14 09:49:44,628][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,628][INFO ][tcp-comm-worker-#26392%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Pinging node: 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,628][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,629][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,629][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, res=false, time=5ms] [2018-11-14 09:49:44,629][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,629][INFO ][tcp-comm-worker-#26392%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished node ping [nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, res=false, time=5ms] [2018-11-14 09:49:44,630][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest4 >>> Grid uptime: 00:00:00.459 [2018-11-14 09:49:44,629][INFO ][tcp-comm-worker-#26432%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Pinging node: 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,631][INFO ][tcp-comm-worker-#26432%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished node ping [nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, res=false, time=7ms] [2018-11-14 09:49:44,629][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 1 ms. [2018-11-14 09:49:44,631][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:44,632][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:44,633][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:44,634][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,634][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,634][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,635][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,635][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=ae293efb-8b41-4b86-b32b-747070800001, evtNodeClient=false] [2018-11-14 09:49:44,634][ERROR][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13188, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=13, nodeId8=3dc77d76, msg=Node joined: TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_JOINED, tstamp=1542188984461], nodeId=99dc07b8, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1542188984379, nodeOrder=0], super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=578, size=245], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=983, size=255], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=179, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=1, size=179], 5017e3fa-a0f1-4c55-ae20-110197800004=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=458, size=272], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=624, size=252], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=184, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=1, size=184]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=113, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=59, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=100, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=1, size=100], 5017e3fa-a0f1-4c55-ae20-110197800004=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=107, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=12, minorTopVer=0], updateSeq=119, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=100, top=AffinityTopologyVersion [topVer=-1, minorTopVer=0], updateSeq=1, size=100]}}, partCntrs=null, partCntrs2=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap2@5b0e54d7, partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [], partsToReload=IgniteDhtPartitionsToReloadMap [], partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], errs=null, compress=true, resTopVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], partCnt=2, super=]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=[/127.0.0.1:45185]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1167) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.lambda$sendAllPartitions$8(GridDhtPartitionsExchangeFuture.java:1957) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendAllPartitions(GridDhtPartitionsExchangeFuture.java:1952) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3383) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processMergedMessage(GridDhtPartitionsExchangeFuture.java:2476) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.fastReplyOnSingleMessage(GridDhtPartitionsExchangeFuture.java:2606) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:342) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:335) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3194) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3173) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:100) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:294) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1568) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1196) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45185, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3461) ... 40 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3298) ... 40 more [2018-11-14 09:49:44,641][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,650][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Config URL: n/a [2018-11-14 09:49:44,636][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 0 ms. [2018-11-14 09:49:44,650][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=140b35b2-0d90-40b8-913e-004ab7400004, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@28eb6ce4, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@3f85f965[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4026bc72, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@18bc4532, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@167dd490, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,656][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Daemon mode: off [2018-11-14 09:49:44,657][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,657][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] OS user: teamcity [2018-11-14 09:49:44,636][INFO ][disco-notifier-worker-#98730%tcp.TcpDiscoveryMultiThreadedTest3%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:44,634][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=ae293efb-8b41-4b86-b32b-747070800001, evtNodeClient=false] [2018-11-14 09:49:44,657][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] PID: 18275 [2018-11-14 09:49:44,651][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 16 ms. [2018-11-14 09:49:44,650][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 5017e3fa-a0f1-4c55-ae20-110197800004 [2018-11-14 09:49:44,658][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,658][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,659][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=5017e3fa-a0f1-4c55-ae20-110197800004, res=false, time=13ms] [2018-11-14 09:49:44,659][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:44,659][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], err=null] [2018-11-14 09:49:44,660][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:44,660][INFO ][sys-#98594%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], durationFromInit=185] [2018-11-14 09:49:44,659][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][CacheAffinitySharedManager] Affinity recalculation (on server join) performed in 1 ms. [2018-11-14 09:49:44,661][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=14, minorTopVer=0], force=false, evt=NODE_JOINED, node=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,659][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM total memory: 1.8GB [2018-11-14 09:49:44,662][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 28 ms. [2018-11-14 09:49:44,659][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 26 ms. [2018-11-14 09:49:44,662][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 29 ms. [2018-11-14 09:49:44,663][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], err=null] [2018-11-14 09:49:44,663][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,664][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], durationFromInit=195] [2018-11-14 09:49:44,664][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Fast replied to single message [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], discoEvt=null, nodeId=ae293efb, evt=NODE_JOINED], nodeId=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:44,662][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,664][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=14, minorTopVer=0], force=false, evt=NODE_JOINED, node=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,663][INFO ][disco-notifier-worker-#98729%tcp.TcpDiscoveryMultiThreadedTest1%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate [2018-11-14 09:49:44,662][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:44,666][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], err=null] [2018-11-14 09:49:44,667][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 1 ms. [2018-11-14 09:49:44,667][INFO ][sys-#98523%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], durationFromInit=195] [2018-11-14 09:49:44,665][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,670][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,667][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=14, minorTopVer=0], force=false, evt=NODE_JOINED, node=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,670][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,674][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,675][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,675][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,676][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,676][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,676][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,677][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=13, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=ae293efb-8b41-4b86-b32b-747070800001, evtNodeClient=false] [2018-11-14 09:49:44,678][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,680][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,680][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity initialization on local join performed in 4 ms. [2018-11-14 09:49:44,681][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 5 ms. [2018-11-14 09:49:44,682][INFO ][node-restart-4][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,682][INFO ][node-restart-4][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,682][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,682][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,682][INFO ][node-restart-4][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,683][INFO ][node-restart-4][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,683][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,683][INFO ][node-restart-4][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,683][INFO ][node-restart-4][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,683][INFO ][node-restart-4][IgnitePluginProcessor] [2018-11-14 09:49:44,683][INFO ][node-restart-4][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,680][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=15, locNode=3dc77d76, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,681][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity initialization on local join performed in 4 ms. [2018-11-14 09:49:44,684][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 8 ms. [2018-11-14 09:49:44,684][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,685][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,684][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 3 ms. [2018-11-14 09:49:44,684][WARN ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,684][INFO ][node-restart-4][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45185, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,686][INFO ][disco-event-worker-#98496%tcp.TcpDiscoveryMultiThreadedTest5%][GridDiscoveryManager] Topology snapshot [ver=15, locNode=4ce119dc, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,686][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], err=null] [2018-11-14 09:49:44,687][WARN ][node-restart-4][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,687][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,687][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,690][WARN ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,690][INFO ][disco-event-worker-#98591%tcp.TcpDiscoveryMultiThreadedTest2%][GridDiscoveryManager] Topology snapshot [ver=15, locNode=77abe0df, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,689][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 5 ms. [2018-11-14 09:49:44,691][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], err=null] [2018-11-14 09:49:44,691][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,692][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003 [2018-11-14 09:49:44,692][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003 [2018-11-14 09:49:44,692][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984686, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], durationFromInit=195] [2018-11-14 09:49:44,688][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=ae293efb-8b41-4b86-b32b-747070800001 [2018-11-14 09:49:44,693][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=14, minorTopVer=0], force=false, evt=NODE_JOINED, node=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:44,691][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=8, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,690][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,694][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,694][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,693][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=ae293efb-8b41-4b86-b32b-747070800001 [2018-11-14 09:49:44,696][WARN ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,696][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=15, locNode=99dc07b8, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,697][WARN ][node-restart-4][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,697][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,696][INFO ][sys-#98733%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:44,696][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ae293efb-8b41-4b86-b32b-747070800001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984686, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], durationFromInit=92] [2018-11-14 09:49:44,697][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,699][INFO ][node-restart-4][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,696][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=14, minorTopVer=0], force=false, evt=NODE_JOINED, node=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:44,699][INFO ][node-restart-4][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,700][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=1/1, time=13 ms] [2018-11-14 09:49:44,700][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:44,701][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=77abe0df-a676-4745-b311-5aa4bb000002, partitionsCount=47, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,701][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=41, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,701][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=5017e3fa-a0f1-4c55-ae20-110197800004, partitionsCount=41, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,699][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,699][INFO ][utility-#98781%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=99dc07b8-e8a8-49ca-bcf9-392b48e00003, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,698][WARN ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,703][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=15, locNode=ae293efb, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,704][ERROR][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=T6 [topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0], topicOrd=-1, ordered=true, timeout=10000, skipOnTimeout=false, msg=GridCacheGroupIdMessage [grpId=1544803905]GridDhtPartitionDemandMessage [rebalanceId=1, parts=IgniteDhtDemandedPartitionsMap [historical=null], timeout=10000, workerId=-1, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], partCnt=41, super=]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3269) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1765) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1228) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.lambda$null$2(GridDhtPartitionDemander.java:514) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6872) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2018-11-14 09:49:44,703][INFO ][utility-#98781%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=99dc07b8-e8a8-49ca-bcf9-392b48e00003, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,703][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, partitionsCount=50, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,703][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[48, 103-104, 201, 221, 223, 272, 287, 298, 303, 308, 340, 356, 413, 419, 463, 490, 542, 545, 555, 581, 612, 639, 643, 658, 660, 672, 675, 705, 819, 846, 848, 873, 890, 922, 927, 949, 1003, 1006-1007, 1010], histPartitions=[]] [2018-11-14 09:49:44,702][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:44,701][INFO ][grid-nio-worker-tcp-comm-2-#98570%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45183, rmtAddr=/127.0.0.1:46940] [2018-11-14 09:49:44,699][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,709][INFO ][grid-nio-worker-tcp-comm-1-#98708%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:46940, rmtAddr=/127.0.0.1:45183] [2018-11-14 09:49:44,709][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] Data Regions Configured: [2018-11-14 09:49:44,708][INFO ][grid-nio-worker-tcp-comm-2-#98426%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45184, rmtAddr=/127.0.0.1:58716] [2018-11-14 09:49:44,708][INFO ][utility-#98784%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=1/4, time=0 ms] [2018-11-14 09:49:44,708][WARN ][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Failed to send initial demand request to node. Failed to send message (node left topology): TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,705][WARN ][node-restart-5][TcpDiscoverySpi] Simulating node failure: 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,704][WARN ][node-restart-2][TcpDiscoverySpi] Simulating node failure: 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,703][INFO ][node-restart-4][ClientListenerProcessor] Client connector processor has started on TCP port 10805 [2018-11-14 09:49:44,712][ERROR][tcp-disco-srvr-#26406%tcp.TcpDiscoveryMultiThreadedTest5%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,711][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=2/4, time=10 ms] [2018-11-14 09:49:44,711][INFO ][sys-#98768%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topic=0, fullPartitions=[66, 74, 92, 99, 113, 120, 143, 145-146, 167, 183, 227, 326-327, 344, 350, 386, 431, 447, 491, 532, 539, 549, 552, 617, 624, 627, 674, 679, 687, 691, 743, 767, 775, 822, 830, 853, 862, 875-876, 880, 911, 915, 941, 954, 972, 1008], histPartitions=[]] [2018-11-14 09:49:44,711][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=1/1, time=10 ms] [2018-11-14 09:49:44,711][INFO ][grid-nio-worker-tcp-comm-2-#98709%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:58716, rmtAddr=/127.0.0.1:45184] [2018-11-14 09:49:44,710][INFO ][utility-#98532%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=ae293efb-8b41-4b86-b32b-747070800001, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,710][INFO ][sys-#98596%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=99dc07b8-e8a8-49ca-bcf9-392b48e00003, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,710][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:44,710][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:44,710][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=5017e3fa-a0f1-4c55-ae20-110197800004, customEvt=null, allowMerge=true] [2018-11-14 09:49:44,716][INFO ][node-restart-3][TcpDiscoveryMultiThreadedTest3] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest3 >>> Local node [ID=99DC07B8-E8A8-49CA-BCF9-392B48E00003, order=13, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10801 TCP:45182 TCP:47503 [2018-11-14 09:49:44,715][INFO ][sys-#98526%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=99dc07b8-e8a8-49ca-bcf9-392b48e00003, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,715][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,718][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,714][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:44,714][INFO ][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Cancelled rebalancing from all nodes [grp=default, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,719][INFO ][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], rebalanceId=1, routines=4] [2018-11-14 09:49:44,719][INFO ][node-restart-4][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,719][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, topic=0, fullPartitions=[0, 12, 14, 65, 77, 84, 96, 123, 147, 163, 180, 203, 212, 231-232, 259, 309, 320, 342, 347, 354-355, 361, 379, 388, 453, 459, 485, 487, 502, 533, 536, 561, 651, 661, 663, 665, 673, 689, 699, 745, 749, 843, 856, 864, 891, 899, 936, 940, 990], histPartitions=[]] [2018-11-14 09:49:44,713][INFO ][tcp-disco-sock-reader-#26449%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:43060, rmtPort=43060 [2018-11-14 09:49:44,712][ERROR][tcp-disco-srvr-#26435%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly.]] java.lang.IllegalStateException: Worker tcp-disco-srvr is terminated unexpectedly. at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5895) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5761) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,721][ERROR][tcp-disco-msg-worker-#26405%tcp.TcpDiscoveryMultiThreadedTest5%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,720][INFO ][node-restart-4][TcpDiscoverySpi] Successfully bound to TCP port [port=47501, localHost=/127.0.0.1, locNodeId=140b35b2-0d90-40b8-913e-004ab7400004] [2018-11-14 09:49:44,720][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [2018-11-14 09:49:44,724][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] Data Regions Configured: [2018-11-14 09:49:44,725][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] ^-- default [initSize=256.0 MiB, maxSize=18.9 GiB, persistence=false] [2018-11-14 09:49:44,725][WARN ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Stop waiting for partitions release latch: Got interrupted while waiting for future to complete. [2018-11-14 09:49:44,718][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=77abe0df-a676-4745-b311-5aa4bb000002, partitionsCount=44, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,717][INFO ][node-restart-3][GridDiscoveryManager] Topology snapshot [ver=13, locNode=99dc07b8, servers=5, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,726][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=49, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,725][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,725][INFO ][node-restart-1][TcpDiscoveryMultiThreadedTest1] >>> +-------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 >>> +-------------------------------------------------------------------------------+ >>> OS name: Linux 4.4.0-134-generic amd64 >>> CPU(s): 5 >>> Heap: 2.0GB >>> VM name: 18275@617cc64df057 >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest1 >>> Local node [ID=AE293EFB-8B41-4B86-B32B-747070800001, order=14, clientMode=false] >>> Local node addresses: [127.0.0.1] >>> Local ports: TCP:10803 TCP:45181 TCP:47502 [2018-11-14 09:49:44,723][INFO ][tcp-disco-sock-reader-#26440%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:47911, rmtPort=47911 [2018-11-14 09:49:44,727][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=52343] [2018-11-14 09:49:44,727][INFO ][node-restart-1][GridDiscoveryManager] Topology snapshot [ver=14, locNode=ae293efb, servers=6, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:44,729][INFO ][sys-#98292%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=ae293efb-8b41-4b86-b32b-747070800001, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,727][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,726][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=5017e3fa-a0f1-4c55-ae20-110197800004, partitionsCount=54, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,726][INFO ][grid-nio-worker-tcp-comm-3-#98571%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45183, rmtAddr=/127.0.0.1:46944] [2018-11-14 09:49:44,726][INFO ][node-restart-3][root] Node started with the following configuration [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@6a3690db], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:44,731][INFO ][grid-nio-worker-tcp-comm-1-#98704%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:46944, rmtAddr=/127.0.0.1:45183] [2018-11-14 09:49:44,730][ERROR][sys-#98764%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=T6 [topic=TOPIC_CACHE, id1=83e8ca36-2305-3266-8e65-1463be879baa, id2=0], topicOrd=-1, ordered=true, timeout=10000, skipOnTimeout=false, msg=GridCacheGroupIdMessage [grpId=1544803905]GridDhtPartitionDemandMessage [rebalanceId=1, parts=IgniteDhtDemandedPartitionsMap [historical=null], timeout=10000, workerId=-1, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], partCnt=54, super=]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3269) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1765) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendOrderedMessage(GridCacheIoManager.java:1228) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemander.lambda$null$2(GridDhtPartitionDemander.java:514) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6872) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2018-11-14 09:49:44,730][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Prepared rebalancing [grp=default, mode=ASYNC, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, partitionsCount=37, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], parallelism=1] [2018-11-14 09:49:44,730][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[44, 51, 95, 98, 131, 156, 174, 190, 193, 262, 288, 294, 296, 318-319, 321-322, 338-339, 346, 348, 406, 409, 420, 458, 473, 488, 497, 538, 569, 585, 601, 611, 644, 671, 712-713, 720, 748, 780, 793, 847, 889, 896, 924, 953, 955, 970, 975], histPartitions=[]] [2018-11-14 09:49:44,730][INFO ][exchange-worker-#98497%tcp.TcpDiscoveryMultiThreadedTest5%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] [2018-11-14 09:49:44,729][INFO ][node-restart-1][root] Node started with the following configuration [id=ae293efb-8b41-4b86-b32b-747070800001, marshaller=BinaryMarshaller [], discovery=TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@179acb48], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], binaryCfg=null, lateAff=true] [2018-11-14 09:49:44,729][ERROR][tcp-disco-msg-worker-#26434%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.InterruptedException]] java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2699) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7117) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2018-11-14 09:49:44,728][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=52343] [2018-11-14 09:49:44,736][INFO ][node-restart-5][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=null, err=class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is stopping: tcp.TcpDiscoveryMultiThreadedTest5] [2018-11-14 09:49:44,736][INFO ][grid-nio-worker-tcp-comm-3-#98427%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45184, rmtAddr=/127.0.0.1:58720] [2018-11-14 09:49:44,735][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:44,735][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=1/4, time=20 ms] [2018-11-14 09:49:44,735][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=ClusterNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, order=5, addr=[127.0.0.1], daemon=false]]] [2018-11-14 09:49:44,735][WARN ][sys-#98764%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Failed to send initial demand request to node. Failed to send message (node left topology): TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:44,732][INFO ][sys-#98600%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=ae293efb-8b41-4b86-b32b-747070800001, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,744][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,742][WARN ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Stop waiting for partitions release latch: Got interrupted while waiting for future to complete. [2018-11-14 09:49:44,740][INFO ][utility-#98793%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalancing [grp=default, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], progress=2/4, time=30 ms] [2018-11-14 09:49:44,740][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]] [2018-11-14 09:49:44,740][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]] [2018-11-14 09:49:44,747][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,740][INFO ][sys-#98744%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=77abe0df-a676-4745-b311-5aa4bb000002, topic=0, fullPartitions=[6, 27, 37, 60, 67, 76, 118, 139, 219, 230, 240, 254-255, 362, 369, 474, 509, 520, 527, 541, 604, 630, 650, 677, 680, 688, 693, 702, 724-725, 736, 750, 768-769, 771, 795, 805, 817, 859, 885, 895, 908, 916, 1009], histPartitions=[]] [2018-11-14 09:49:44,740][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]]] [2018-11-14 09:49:44,739][INFO ][grid-nio-worker-tcp-comm-2-#98705%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:58720, rmtAddr=/127.0.0.1:45184] [2018-11-14 09:49:44,739][INFO ][node-restart-5][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=null, durationFromInit=43] [2018-11-14 09:49:44,739][INFO ][tcp-disco-sock-reader-#26472%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:52343, rmtPort=52343] [2018-11-14 09:49:44,749][INFO ][sys-#98513%tcp.TcpDiscoveryMultiThreadedTest5%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=default, demander=ae293efb-8b41-4b86-b32b-747070800001, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], topic=0] [2018-11-14 09:49:44,749][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,748][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,751][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] [2018-11-14 09:49:44,747][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,746][INFO ][sys-#98764%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Cancelled rebalancing from all nodes [grp=default, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]] [2018-11-14 09:49:44,752][INFO ][sys-#98764%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0], rebalanceId=1, routines=4] [2018-11-14 09:49:44,746][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:44,745][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60865] [2018-11-14 09:49:44,752][INFO ][sys-#98761%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionDemander] Started rebalance routine [default, supplier=4ce119dc-4ac4-43db-b3ca-aba085b00005, topic=0, fullPartitions=[75, 83, 112, 125, 200, 208, 299, 323, 325, 351, 374, 400, 424, 466, 477, 523, 543, 547-548, 563, 579, 596, 621-622, 676, 778, 791, 806, 860, 887, 903, 919, 933, 979, 982, 987, 996], histPartitions=[]] [2018-11-14 09:49:44,752][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,751][INFO ][node-restart-5][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:44,751][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=15, minorTopVer=0], node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, allReceived=false] [2018-11-14 09:49:44,754][INFO ][node-restart-5][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:44,754][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] [2018-11-14 09:49:44,755][INFO ][node-restart-5][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983463, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=f97f2a11761-4ce119dc-4ac4-43db-b3ca-aba085b00005, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:44,753][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:44,753][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60865] [2018-11-14 09:49:44,756][INFO ][exchange-worker-#98592%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=false] [2018-11-14 09:49:44,754][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=15, minorTopVer=0], node=77abe0df-a676-4745-b311-5aa4bb000002, allReceived=false] [2018-11-14 09:49:44,757][INFO ][sys-#98290%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=15, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001, allReceived=false] [2018-11-14 09:49:44,757][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=15, minorTopVer=0], node=4ce119dc-4ac4-43db-b3ca-aba085b00005, allReceived=true] [2018-11-14 09:49:44,758][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=15, minorTopVer=0]] [2018-11-14 09:49:44,758][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:44,758][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] [2018-11-14 09:49:44,756][INFO ][node-restart-2][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=null, err=class org.apache.ignite.internal.IgniteInterruptedCheckedException: Node is stopping: tcp.TcpDiscoveryMultiThreadedTest2] [2018-11-14 09:49:44,759][INFO ][node-restart-2][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=77abe0df-a676-4745-b311-5aa4bb000002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984074, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=null, durationFromInit=63] [2018-11-14 09:49:44,760][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 2 ms. [2018-11-14 09:49:44,756][INFO ][tcp-disco-sock-reader-#26473%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60865, rmtPort=60865] [2018-11-14 09:49:44,760][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:44,760][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:44,761][INFO ][node-restart-2][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983770, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=22ef2a11761-77abe0df-a676-4745-b311-5aa4bb000002, userVer=0, loc=true, sampleClsName=org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:44,760][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest5 >>> Grid uptime: 00:00:01.235 [2018-11-14 09:49:44,758][INFO ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Pinging node: 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,763][INFO ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished node ping [nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, res=false, time=14ms] [2018-11-14 09:49:44,763][WARN ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,758][WARN ][tcp-comm-worker-#26450%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Failed to connect to a remote node (make sure that destination node is alive and operating system firewall is disabled on local and remote hosts) [addrs=[/127.0.0.1:45184]] [2018-11-14 09:49:44,757][WARN ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Connect timed out (consider increasing 'failureDetectionTimeout' configuration property) [addr=/127.0.0.1:45184, failureDetectionTimeout=10000] [2018-11-14 09:49:44,764][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,764][INFO ][tcp-comm-worker-#26450%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Pinging node: 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,765][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, res=false, time=6ms] [2018-11-14 09:49:44,760][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:44,765][INFO ][tcp-comm-worker-#26450%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished node ping [nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, res=false, time=6ms] [2018-11-14 09:49:44,765][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest2 >>> Grid uptime: 00:00:00.939 [2018-11-14 09:49:44,765][INFO ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Pinging node: 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,766][INFO ][tcp-comm-worker-#26450%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Pinging node: 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,770][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53424] [2018-11-14 09:49:44,766][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 1 ms. [2018-11-14 09:49:44,771][WARN ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[17-19, 22-23, 26, 29, 32, 36, 39-40, 42, 45-46, 49, 56-59, 61-63, 69, 71, 73, 79-80, 86, 88, 90, 94, 101, 105, 107, 109, 114, 117, 121, 130, 133, 136-138, 140-141, 144, 153, 155, 157, 161-162, 166, 169-170, 173, 175, 177-179, 184, 187-188, 192, 194, 198, 204, 209, 211, 213, 215, 233, 235-236, 238, 243-244, 246, 249-250, 252-253, 256-258, 260, 264-265, 270, 273, 276-278, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 328, 330, 332-333, 337, 341, 343, 349, 352, 357, 363, 373, 382-383, 389, 392, 401-402, 407-408, 422-423, 425-426, 429-430, 432, 440, 443-444, 454, 464, 467, 469, 480, 482-483, 492, 496, 498, 503, 506, 512, 515-517, 521, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 570, 575-578, 580, 587-588, 597, 599-600, 602, 613, 618, 620, 637, 641-642, 655, 657, 662, 668, 678, 682, 686, 695-696, 698, 700-701, 704, 706, 710, 716-719, 722, 730, 734, 737, 741-742, 744, 746-747, 751-752, 754, 757, 759, 762, 772-774, 776-777, 788, 802, 807-808, 811-812, 814, 818, 821, 823, 827-829, 832, 834, 849, 854, 869, 872, 892, 905, 912, 914, 920, 925, 930-931, 939, 943, 946, 948, 956-957, 959, 964, 966-967, 969, 976-977, 984, 986, 989, 991, 993-995, 1004-1005, 1012-1013, 1015, 1017, 1022], plc=IGNORE] [2018-11-14 09:49:44,765][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,772][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:44,771][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53424] [2018-11-14 09:49:44,771][INFO ][tcp-disco-sock-reader-#26473%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188984769, connCheckInterval=500] [2018-11-14 09:49:44,774][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:44,774][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:44,774][INFO ][tcp-disco-sock-reader-#26473%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:60865, rmtPort=60865 [2018-11-14 09:49:44,776][INFO ][tcp-disco-sock-reader-#26474%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:53424, rmtPort=53424] [2018-11-14 09:49:44,776][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=35376] [2018-11-14 09:49:44,776][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,775][ERROR][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13221, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=15, nodeId8=3dc77d76, msg=Node failed: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_FAILED, tstamp=1542188984676], nodeId=5017e3fa, evt=NODE_FAILED], lastVer=GridCacheVersion [topVer=0, order=1542188984379, nodeOrder=0], super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1250, size=296], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap2@19c8614d, partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [], partsToReload=IgniteDhtPartitionsToReloadMap [], partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], partCnt=2, super=]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=[/127.0.0.1:45184]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1167) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.lambda$sendAllPartitions$8(GridDhtPartitionsExchangeFuture.java:1957) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendAllPartitions(GridDhtPartitionsExchangeFuture.java:1952) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3383) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3157) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:2773) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:139) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2578) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2566) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:395) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:507) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:486) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:463) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initDone(GridDhtPartitionsExchangeFuture.java:4181) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1469) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:813) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2759) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45184, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3461) ... 39 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3298) ... 39 more [2018-11-14 09:49:44,775][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,791][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Config URL: n/a [2018-11-14 09:49:44,791][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 4ce119dc-4ac4-43db-b3ca-aba085b00005 [2018-11-14 09:49:44,791][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=d0e6867e-ed6d-47a4-baf4-96cdfbd00005, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@7e27330d, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@6f5f84a0[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@4a77d8, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@60041246, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@241c33b3, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,797][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Daemon mode: off [2018-11-14 09:49:44,775][INFO ][tcp-disco-sock-reader-#26472%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:52343, rmtPort=52343 [2018-11-14 09:49:44,792][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=4ce119dc-4ac4-43db-b3ca-aba085b00005, res=false, time=6ms] [2018-11-14 09:49:44,792][INFO ][tcp-disco-sock-reader-#26474%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=ae293efb-8b41-4b86-b32b-747070800001, rmtAddr=/127.0.0.1:53424, rmtPort=53424] [2018-11-14 09:49:44,786][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=35376] [2018-11-14 09:49:44,786][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=52964] [2018-11-14 09:49:44,798][ERROR][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13221, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=15, nodeId8=3dc77d76, msg=Node failed: TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_FAILED, tstamp=1542188984676], nodeId=5017e3fa, evt=NODE_FAILED], lastVer=GridCacheVersion [topVer=0, order=1542188984379, nodeOrder=0], super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1250, size=296], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionCountersMap2@19c8614d, partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [], partsToReload=IgniteDhtPartitionsToReloadMap [], partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], partCnt=2, super=]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, addrs=[/127.0.0.1:45183]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.send(GridCacheIoManager.java:1167) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.lambda$sendAllPartitions$8(GridDhtPartitionsExchangeFuture.java:1957) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.Iterator.forEachRemaining(Iterator.java:116) at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.sendAllPartitions(GridDhtPartitionsExchangeFuture.java:1952) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3383) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3157) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:2773) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:139) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2578) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2566) at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:395) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:349) at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:337) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:507) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:486) at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:463) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.initDone(GridDhtPartitionsExchangeFuture.java:4181) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1469) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:813) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2759) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45183, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3461) ... 39 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3298) ... 39 more [2018-11-14 09:49:44,797][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,812][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] OS user: teamcity [2018-11-14 09:49:44,812][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] PID: 18275 [2018-11-14 09:49:44,812][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Pinging node: 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,799][INFO ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished node ping [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, res=false, time=40ms] [2018-11-14 09:49:44,799][INFO ][tcp-disco-sock-reader-#26474%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=ae293efb-8b41-4b86-b32b-747070800001, rmtAddr=/127.0.0.1:53424, rmtPort=53424] [2018-11-14 09:49:44,813][INFO ][tcp-disco-sock-reader-#26474%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:53424, rmtPort=53424 [2018-11-14 09:49:44,813][WARN ][tcp-comm-worker-#26452%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Failed to ping node (status check will be initiated): 77abe0df-a676-4745-b311-5aa4bb000002 [2018-11-14 09:49:44,812][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=52964] [2018-11-14 09:49:44,812][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=50486] [2018-11-14 09:49:44,812][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,815][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,815][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM total memory: 1.8GB [2018-11-14 09:49:44,815][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,814][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=50486] [2018-11-14 09:49:44,814][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327e >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [2018-11-14 09:49:44,816][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Config URL: n/a [2018-11-14 09:49:44,814][INFO ][tcp-disco-sock-reader-#26475%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:35376, rmtPort=35376] [2018-11-14 09:49:44,817][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IgniteConfiguration [igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=5, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, igniteHome=/data/teamcity/work/ae48a463bd611b52, igniteWorkDir=/data/teamcity/work/ae48a463bd611b52/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1866da85, nodeId=b1b3ec2d-cc09-4372-90a8-57d54c800002, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=10000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=1000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@5e000d9b, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=45180, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@7c891f83[Count = 1], stopping=false], evtSpi=MemoryEventStorageSpi [expireAgeMs=9223372036854775807, expireCnt=10000, filter=null], colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@5a723ff9, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5301bab8, clientMode=false, rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0.1, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=10000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=null, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=20268891340, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=false, checkpointPageBufSize=0], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@195fa2d3, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]NoOpFailureHandler [super=], commFailureRslvr=null] [2018-11-14 09:49:44,823][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Daemon mode: off [2018-11-14 09:49:44,823][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS: Linux 4.4.0-134-generic amd64 [2018-11-14 09:49:44,824][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] OS user: teamcity [2018-11-14 09:49:44,816][INFO ][tcp-disco-sock-reader-#26476%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:52964, rmtPort=52964] [2018-11-14 09:49:44,815][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,824][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,824][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53566] [2018-11-14 09:49:44,824][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] PID: 18275 [2018-11-14 09:49:44,822][INFO ][tcp-disco-sock-reader-#26477%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:50486, rmtPort=50486] [2018-11-14 09:49:44,817][INFO ][tcp-disco-sock-reader-#26475%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, rmtAddr=/127.0.0.1:35376, rmtPort=35376] [2018-11-14 09:49:44,825][INFO ][tcp-disco-sock-reader-#26477%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Received ping request from the remote node [rmtNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, rmtAddr=/127.0.0.1:50486, rmtPort=50486] [2018-11-14 09:49:44,826][INFO ][tcp-comm-worker-#26450%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished node ping [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, res=false, time=67ms] [2018-11-14 09:49:44,825][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Language runtime: Java Platform API Specification ver. 1.8 [2018-11-14 09:49:44,826][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM information: Java(TM) SE Runtime Environment 1.8.0_181-b13 Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 25.181-b13 [2018-11-14 09:49:44,827][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM total memory: 1.8GB [2018-11-14 09:49:44,827][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Remote Management [restart: off, REST: off, JMX (remote: off)] [2018-11-14 09:49:44,825][INFO ][tcp-disco-sock-reader-#26476%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Connection check done: [liveAddr=/127.0.0.1:47501, previousNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47501], connectingNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,825][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53566] [2018-11-14 09:49:44,824][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,831][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,828][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Logger: GridTestLog4jLogger [quiet=false, config=null] [2018-11-14 09:49:44,832][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] IGNITE_HOME=/data/teamcity/work/ae48a463bd611b52 [2018-11-14 09:49:44,832][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] VM arguments: [-DIGNITE_DUMP_THREADS_ON_FAILURE=false, -DMono=/usr/bin/mono-sgen, -DMono4.0_x64=/usr/lib/mono/4.0, -DMono4.5_x64=/usr/lib/mono/4.5, -DMonoVersion=5.12.0.226, -Dagent.home.dir=/home/teamcity/agent, -Dagent.name=publicagent11_9091, -Dagent.ownPort=9091, -Dagent.work.dir=/data/teamcity/work, -Dbuild.number=5235, -Dbuild.vcs.number=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.1=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dbuild.vcs.number.GitHubApacheIgnite=b186327eed79f28ce0d5f49cb6f1ee978c89de94, -Dclassworlds.conf=/data/teamcity/temp/buildTmp/teamcity.m2.conf, -Dcom.jetbrains.maven.watcher.report.file=/data/teamcity/temp/buildTmp/maven-build-info.xml, -Djava.io.tmpdir=/data/teamcity/temp/buildTmp, -Dmaven.home=/home/teamcity/agent/tools/maven.3.5.3, -Dmaven.multiModuleProjectDirectory=/data/teamcity/work/ae48a463bd611b52, -Dmaven.repo.local=/home/teamcity/.m2/repository, -Dteamcity.agent.cpuBenchmark=530, -Dteamcity.agent.dotnet.agent_url=http://localhost:9091/RPC2, -Dteamcity.agent.dotnet.build_id=2315054, -Dteamcity.auth.password=JRDOKUzcT4F3qZQfPBlA9KxlqkPhphbu, -Dteamcity.auth.userId=TeamCityBuildId=2315054, -Dteamcity.build.changedFiles.file=/data/teamcity/temp/buildTmp/changedFiles7532591010222528438.txt, -Dteamcity.build.checkoutDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.build.id=2315054, -Dteamcity.build.properties.file=/data/teamcity/temp/buildTmp/teamcity.build4865164388563229832.properties, -Dteamcity.build.tempDir=/data/teamcity/temp/buildTmp, -Dteamcity.build.workingDir=/data/teamcity/work/ae48a463bd611b52, -Dteamcity.buildConfName=SPI, -Dteamcity.buildType.id=IgniteTests24Java8_Spi, -Dteamcity.configuration.properties.file=/data/teamcity/temp/buildTmp/teamcity.config3962743277867118515.properties, -Dteamcity.maven.watcher.home=/home/teamcity/agent/plugins/mavenPlugin/maven-watcher-jdk16, -Dteamcity.projectName=Ignite Tests 2.4+ (Java 8/9/10/11), -Dteamcity.runner.properties.file=/data/teamcity/temp/buildTmp/teamcity.runner7316552539640129396.properties, -Dteamcity.tests.recentlyFailedTests.file=/data/teamcity/temp/buildTmp/testsToRunFirst812558998635982608.txt, -Dteamcity.version=2017.2.3 (build 51047), -ea, -Xms2g, -Xmx2g, -XX:+HeapDumpOnOutOfMemoryError, -XX:+AggressiveOpts, -DIGNITE_TEST_HOME="/data/teamcity/work/ae48a463bd611b52", -DIGNITE_UPDATE_NOTIFIER=false, -DIGNITE_NO_DISCO_ORDER=true, -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true, -DIGNITE_QUIET=false, -Djava.net.preferIPv4Stack=true] [2018-11-14 09:49:44,835][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [2018-11-14 09:49:44,835][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,835][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,828][INFO ][tcp-disco-sock-reader-#26476%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=true, lastMessageReceivedTime=1542188984699, now=1542188984816, connCheckInterval=500] [2018-11-14 09:49:44,827][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, res=false, time=21ms] [2018-11-14 09:49:44,826][INFO ][tcp-comm-worker-#26381%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished node ping [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, res=false, time=67ms] [2018-11-14 09:49:44,826][INFO ][tcp-disco-sock-reader-#26477%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, rmtAddr=/127.0.0.1:50486, rmtPort=50486] [2018-11-14 09:49:44,837][INFO ][tcp-disco-sock-reader-#26477%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:50486, rmtPort=50486 [2018-11-14 09:49:44,826][INFO ][tcp-disco-sock-reader-#26475%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished writing ping response [rmtNodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, rmtAddr=/127.0.0.1:35376, rmtPort=35376] [2018-11-14 09:49:44,837][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] [2018-11-14 09:49:44,837][INFO ][tcp-disco-sock-reader-#26476%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:52964, rmtPort=52964 [2018-11-14 09:49:44,836][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node has connection to it's previous, trying previous again. [next=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,836][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 62 ms. [2018-11-14 09:49:44,839][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 65 ms. [2018-11-14 09:49:44,839][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], err=null] [2018-11-14 09:49:44,839][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:44,836][INFO ][tcp-disco-sock-reader-#26478%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:53566, rmtPort=53566] [2018-11-14 09:49:44,832][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [2018-11-14 09:49:44,840][WARN ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [2018-11-14 09:49:44,840][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984054, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], durationFromInit=161] [2018-11-14 09:49:44,838][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:44,838][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:44,842][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=46507] [2018-11-14 09:49:44,842][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 4 ms. [2018-11-14 09:49:44,837][INFO ][tcp-disco-sock-reader-#26475%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:35376, rmtPort=35376 [2018-11-14 09:49:44,837][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0]] [2018-11-14 09:49:44,843][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,842][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=46507] [2018-11-14 09:49:44,841][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], crd=true] [2018-11-14 09:49:44,840][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] 3-rd party licenses can be found at: /data/teamcity/work/ae48a463bd611b52/libs/licenses [2018-11-14 09:49:44,846][INFO ][node-restart-5][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,846][INFO ][node-restart-5][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,846][INFO ][node-restart-5][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,847][INFO ][node-restart-5][IgnitePluginProcessor] [2018-11-14 09:49:44,840][WARN ][tcp-disco-sock-reader-#26478%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Failed to read magic header (too few bytes received) [rmtAddr=/127.0.0.1:53566, locAddr=/127.0.0.1:47501] [2018-11-14 09:49:44,847][INFO ][node-restart-5][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,847][ERROR][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13222, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=]]] class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, addrs=[/127.0.0.1:45183]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2858) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45183, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3461) ... 14 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3298) ... 14 more [2018-11-14 09:49:44,847][INFO ][node-restart-2][IgnitePluginProcessor] Configured plugins: [2018-11-14 09:49:44,844][INFO ][tcp-disco-sock-reader-#26479%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:46507, rmtPort=46507] [2018-11-14 09:49:44,844][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:44,856][INFO ][tcp-disco-sock-reader-#26479%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188984847, connCheckInterval=500] [2018-11-14 09:49:44,857][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 14 ms. [2018-11-14 09:49:44,844][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 2 ms. [2018-11-14 09:49:44,857][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], err=null] [2018-11-14 09:49:44,858][WARN ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[17-19, 22-23, 26, 29, 32, 36, 39-40, 42, 45-46, 49, 56-59, 61-63, 69, 71, 73, 79-80, 86, 88, 90, 94, 101, 105, 107, 109, 114, 117, 121, 130, 133, 136-138, 140-141, 144, 153, 155, 157, 161-162, 166, 169-170, 173, 175, 177-179, 184, 187-188, 192, 194, 198, 204, 209, 211, 213, 215, 233, 235-236, 238, 243-244, 246, 249-250, 252-253, 256-258, 260, 264-265, 270, 273, 276-278, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 328, 330, 332-333, 337, 341, 343, 349, 352, 357, 363, 373, 382-383, 389, 392, 401-402, 407-408, 422-423, 425-426, 429-430, 432, 440, 443-444, 454, 464, 467, 469, 480, 482-483, 492, 496, 498, 503, 506, 512, 515-517, 521, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 570, 575-578, 580, 587-588, 597, 599-600, 602, 613, 618, 620, 637, 641-642, 655, 657, 662, 668, 678, 682, 686, 695-696, 698, 700-701, 704, 706, 710, 716-719, 722, 730, 734, 737, 741-742, 744, 746-747, 751-752, 754, 757, 759, 762, 772-774, 776-777, 788, 802, 807-808, 811-812, 814, 818, 821, 823, 827-829, 832, 834, 849, 854, 869, 872, 892, 905, 912, 914, 920, 925, 930-931, 939, 943, 946, 948, 956-957, 959, 964, 966-967, 969, 976-977, 984, 986, 989, 991, 993-995, 1004-1005, 1012-1013, 1015, 1017, 1022], plc=IGNORE] [2018-11-14 09:49:44,855][INFO ][node-restart-2][IgnitePluginProcessor] ^-- TestReconnectPlugin 1.0 [2018-11-14 09:49:44,862][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,862][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:44,862][INFO ][node-restart-2][IgnitePluginProcessor] ^-- StanByClusterTestProvider 1.0 [2018-11-14 09:49:44,863][INFO ][node-restart-2][IgnitePluginProcessor] ^-- null [2018-11-14 09:49:44,863][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:44,863][INFO ][node-restart-2][IgnitePluginProcessor] ^-- PageMemory tracker plugin 1.0 [2018-11-14 09:49:44,863][INFO ][node-restart-2][IgnitePluginProcessor] ^-- [2018-11-14 09:49:44,863][INFO ][node-restart-2][IgnitePluginProcessor] [2018-11-14 09:49:44,849][INFO ][node-restart-5][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45183, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,864][INFO ][tcp-disco-sock-reader-#26479%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:46507, rmtPort=46507 [2018-11-14 09:49:44,848][INFO ][tcp-disco-sock-reader-#26478%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:53566, rmtPort=53566 [2018-11-14 09:49:44,864][WARN ][node-restart-5][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,863][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 6 ms. [2018-11-14 09:49:44,863][INFO ][node-restart-2][FailureProcessor] Configured failure handler: [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=SingletonSet [SYSTEM_WORKER_BLOCKED]]]] [2018-11-14 09:49:44,862][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 5 ms. [2018-11-14 09:49:44,866][INFO ][sys-#98736%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], durationFromInit=171] [2018-11-14 09:49:44,860][WARN ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Failed to send partitions full message [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], err=class o.a.i.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=GridCacheMessage [msgId=13222, depInfo=null, err=null, skipPrepare=false]]], policy=2]] class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left the grid or TCP connection cannot be established due to firewall issues) [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topic=TOPIC_CACHE, msg=GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=GridCacheMessage [msgId=13222, depInfo=null, err=null, skipPrepare=false]]], policy=2] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1666) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2858) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2746) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) ... 9 more Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is node still alive?). Make sure that each ComputeTask and cache Transaction has a timeout set in order to prevent parties from waiting forever in case of network issues [nodeId=77abe0df-a676-4745-b311-5aa4bb000002, addrs=[/127.0.0.1:45183]] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3458) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) ... 11 more Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect to address [addr=/127.0.0.1:45183, err=Connection refused] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3461) ... 14 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3298) ... 14 more [2018-11-14 09:49:44,871][WARN ][node-restart-5][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,867][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=15, minorTopVer=0], force=false, evt=NODE_FAILED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,867][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], err=null] [2018-11-14 09:49:44,866][INFO ][node-restart-2][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=45184, locHost=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] [2018-11-14 09:49:44,881][INFO ][sys-#98744%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Delay process full message without exchange id (there is exchange in progress) [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:44,880][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,881][WARN ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[17-19, 22-23, 26, 29, 32, 36, 39-40, 42, 45-46, 49, 56-59, 61-63, 69, 71, 73, 79-80, 86, 88, 90, 94, 101, 105, 107, 109, 114, 117, 121, 130, 133, 136-138, 140-141, 144, 153, 155, 157, 161-162, 166, 169-170, 173, 175, 177-179, 184, 187-188, 192, 194, 198, 204, 209, 211, 213, 215, 233, 235-236, 238, 243-244, 246, 249-250, 252-253, 256-258, 260, 264-265, 270, 273, 276-278, 281, 285, 290, 292-293, 297, 300, 307, 310, 314-315, 328, 330, 332-333, 337, 341, 343, 349, 352, 357, 363, 373, 382-383, 389, 392, 401-402, 407-408, 422-423, 425-426, 429-430, 432, 440, 443-444, 454, 464, 467, 469, 480, 482-483, 492, 496, 498, 503, 506, 512, 515-517, 521, 526, 530-531, 534-535, 550, 554, 556, 559, 565-566, 570, 575-578, 580, 587-588, 597, 599-600, 602, 613, 618, 620, 637, 641-642, 655, 657, 662, 668, 678, 682, 686, 695-696, 698, 700-701, 704, 706, 710, 716-719, 722, 730, 734, 737, 741-742, 744, 746-747, 751-752, 754, 757, 759, 762, 772-774, 776-777, 788, 802, 807-808, 811-812, 814, 818, 821, 823, 827-829, 832, 834, 849, 854, 869, 872, 892, 905, 912, 914, 920, 925, 930-931, 939, 943, 946, 948, 956-957, 959, 964, 966-967, 969, 976-977, 984, 986, 989, 991, 993-995, 1004-1005, 1012-1013, 1015, 1017, 1022], plc=IGNORE] [2018-11-14 09:49:44,881][WARN ][node-restart-2][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [2018-11-14 09:49:44,884][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 3 ms. [2018-11-14 09:49:44,884][INFO ][grid-nio-worker-tcp-comm-0-#98849%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45184, rmtAddr=/127.0.0.1:58746] [2018-11-14 09:49:44,885][INFO ][sys-#98743%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ae293efb-8b41-4b86-b32b-747070800001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=5017e3fa-a0f1-4c55-ae20-110197800004, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=10, intOrder=8, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], durationFromInit=169] [2018-11-14 09:49:44,886][WARN ][node-restart-2][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [2018-11-14 09:49:44,886][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Security status [authentication=off, tls/ssl=off] [2018-11-14 09:49:44,883][INFO ][node-restart-5][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,886][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=15, minorTopVer=0], force=false, evt=NODE_FAILED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:44,887][INFO ][node-restart-5][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,888][INFO ][node-restart-2][IgniteCacheDatabaseSharedManager] Configured data regions initialized successfully [total=3] [2018-11-14 09:49:44,889][INFO ][node-restart-2][PartitionsEvictManager] Evict partition permits=2 [2018-11-14 09:49:44,891][INFO ][node-restart-2][ClientListenerProcessor] Client connector processor has started on TCP port 10802 [2018-11-14 09:49:44,891][INFO ][node-restart-5][ClientListenerProcessor] Client connector processor has started on TCP port 10804 [2018-11-14 09:49:44,894][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,894][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,894][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Non-loopback local IPs: 172.17.0.2 [2018-11-14 09:49:44,894][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Enabled local MACs: 0242AC110002 [2018-11-14 09:49:44,895][INFO ][node-restart-5][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,895][INFO ][node-restart-2][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [2018-11-14 09:49:44,896][INFO ][node-restart-5][TcpDiscoverySpi] Successfully bound to TCP port [port=47504, localHost=/127.0.0.1, locNodeId=d0e6867e-ed6d-47a4-baf4-96cdfbd00005] [2018-11-14 09:49:44,896][INFO ][node-restart-2][TcpDiscoverySpi] Successfully bound to TCP port [port=47505, localHost=/127.0.0.1, locNodeId=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:44,899][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60564] [2018-11-14 09:49:44,899][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60564] [2018-11-14 09:49:44,900][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=44283] [2018-11-14 09:49:44,900][INFO ][tcp-disco-sock-reader-#26488%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60564, rmtPort=60564] [2018-11-14 09:49:44,900][INFO ][tcp-disco-srvr-#26458%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=44283] [2018-11-14 09:49:44,901][INFO ][tcp-disco-sock-reader-#26489%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:44283, rmtPort=44283] [2018-11-14 09:49:44,902][INFO ][tcp-disco-sock-reader-#26488%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:60564, rmtPort=60564 [2018-11-14 09:49:44,902][INFO ][tcp-disco-sock-reader-#26489%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:44283, rmtPort=44283 [2018-11-14 09:49:44,980][INFO ][grid-nio-worker-tcp-comm-0-#98843%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45183, rmtAddr=/127.0.0.1:46974] [2018-11-14 09:49:45,058][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node, try previous [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,060][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,061][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=49275] [2018-11-14 09:49:45,061][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=49275] [2018-11-14 09:49:45,062][INFO ][tcp-disco-sock-reader-#26490%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:49275, rmtPort=49275] [2018-11-14 09:49:45,063][INFO ][tcp-disco-sock-reader-#26490%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:49275, rmtPort=49275 [2018-11-14 09:49:45,063][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,065][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,066][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=36719] [2018-11-14 09:49:45,066][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=36719] [2018-11-14 09:49:45,066][INFO ][tcp-disco-sock-reader-#26491%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:36719, rmtPort=36719] [2018-11-14 09:49:45,067][INFO ][tcp-disco-sock-reader-#26491%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985059, connCheckInterval=500] [2018-11-14 09:49:45,068][INFO ][tcp-disco-sock-reader-#26491%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:36719, rmtPort=36719 [2018-11-14 09:49:45,068][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,070][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,071][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=37622] [2018-11-14 09:49:45,071][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=37622] [2018-11-14 09:49:45,072][INFO ][tcp-disco-sock-reader-#26492%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:37622, rmtPort=37622] [2018-11-14 09:49:45,072][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53578] [2018-11-14 09:49:45,072][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53578] [2018-11-14 09:49:45,073][INFO ][tcp-disco-sock-reader-#26493%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:53578, rmtPort=53578] [2018-11-14 09:49:45,073][INFO ][tcp-disco-sock-reader-#26492%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Connection check done: [liveAddr=/127.0.0.1:47501, previousNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47501], connectingNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:45,073][WARN ][tcp-disco-sock-reader-#26493%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Failed to read magic header (too few bytes received) [rmtAddr=/127.0.0.1:53578, locAddr=/127.0.0.1:47501] [2018-11-14 09:49:45,074][INFO ][tcp-disco-sock-reader-#26492%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=true, lastMessageReceivedTime=1542188984699, now=1542188985069, connCheckInterval=500] [2018-11-14 09:49:45,074][INFO ][tcp-disco-sock-reader-#26493%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:53578, rmtPort=53578 [2018-11-14 09:49:45,075][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node has connection to it's previous, trying previous again. [next=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,076][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,075][INFO ][tcp-disco-sock-reader-#26492%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:37622, rmtPort=37622 [2018-11-14 09:49:45,077][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=39331] [2018-11-14 09:49:45,077][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=39331] [2018-11-14 09:49:45,077][INFO ][tcp-disco-sock-reader-#26494%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39331, rmtPort=39331] [2018-11-14 09:49:45,078][INFO ][tcp-disco-sock-reader-#26494%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985069, connCheckInterval=500] [2018-11-14 09:49:45,079][INFO ][tcp-disco-sock-reader-#26494%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39331, rmtPort=39331 [2018-11-14 09:49:45,279][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node, try previous [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,281][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,281][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=55067] [2018-11-14 09:49:45,282][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=55067] [2018-11-14 09:49:45,282][INFO ][tcp-disco-sock-reader-#26495%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:55067, rmtPort=55067] [2018-11-14 09:49:45,283][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,283][INFO ][tcp-disco-sock-reader-#26495%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:55067, rmtPort=55067 [2018-11-14 09:49:45,285][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,286][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=56726] [2018-11-14 09:49:45,286][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=56726] [2018-11-14 09:49:45,287][INFO ][tcp-disco-sock-reader-#26496%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:56726, rmtPort=56726] [2018-11-14 09:49:45,287][INFO ][tcp-disco-sock-reader-#26496%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985280, connCheckInterval=500] [2018-11-14 09:49:45,288][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,288][INFO ][tcp-disco-sock-reader-#26496%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:56726, rmtPort=56726 [2018-11-14 09:49:45,290][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,291][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=43339] [2018-11-14 09:49:45,291][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=43339] [2018-11-14 09:49:45,292][INFO ][tcp-disco-sock-reader-#26497%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:43339, rmtPort=43339] [2018-11-14 09:49:45,292][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53580] [2018-11-14 09:49:45,292][INFO ][tcp-disco-sock-reader-#26497%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Connection check done: [liveAddr=/127.0.0.1:47501, previousNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47501], connectingNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:45,293][INFO ][tcp-disco-sock-reader-#26497%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=true, lastMessageReceivedTime=1542188984699, now=1542188985290, connCheckInterval=500] [2018-11-14 09:49:45,292][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53580] [2018-11-14 09:49:45,294][INFO ][tcp-disco-sock-reader-#26498%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:53580, rmtPort=53580] [2018-11-14 09:49:45,294][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node has connection to it's previous, trying previous again. [next=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,294][WARN ][tcp-disco-sock-reader-#26498%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Failed to read magic header (too few bytes received) [rmtAddr=/127.0.0.1:53580, locAddr=/127.0.0.1:47501] [2018-11-14 09:49:45,294][INFO ][tcp-disco-sock-reader-#26497%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:43339, rmtPort=43339 [2018-11-14 09:49:45,295][INFO ][tcp-disco-sock-reader-#26498%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:53580, rmtPort=53580 [2018-11-14 09:49:45,294][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,296][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=48690] [2018-11-14 09:49:45,296][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=48690] [2018-11-14 09:49:45,296][INFO ][tcp-disco-sock-reader-#26499%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:48690, rmtPort=48690] [2018-11-14 09:49:45,297][INFO ][tcp-disco-sock-reader-#26499%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985290, connCheckInterval=500] [2018-11-14 09:49:45,298][INFO ][tcp-disco-sock-reader-#26499%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:48690, rmtPort=48690 [2018-11-14 09:49:45,498][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node, try previous [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,500][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,500][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=40845] [2018-11-14 09:49:45,501][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=40845] [2018-11-14 09:49:45,501][INFO ][tcp-disco-sock-reader-#26500%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:40845, rmtPort=40845] [2018-11-14 09:49:45,502][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,502][INFO ][tcp-disco-sock-reader-#26500%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:40845, rmtPort=40845 [2018-11-14 09:49:45,504][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,505][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=33184] [2018-11-14 09:49:45,505][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=33184] [2018-11-14 09:49:45,506][INFO ][tcp-disco-sock-reader-#26501%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:33184, rmtPort=33184] [2018-11-14 09:49:45,506][INFO ][tcp-disco-sock-reader-#26501%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985502, connCheckInterval=500] [2018-11-14 09:49:45,507][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,507][INFO ][tcp-disco-sock-reader-#26501%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:33184, rmtPort=33184 [2018-11-14 09:49:45,509][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,510][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=57367] [2018-11-14 09:49:45,510][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=57367] [2018-11-14 09:49:45,510][INFO ][tcp-disco-sock-reader-#26502%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:57367, rmtPort=57367] [2018-11-14 09:49:45,511][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=53582] [2018-11-14 09:49:45,511][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=53582] [2018-11-14 09:49:45,511][INFO ][tcp-disco-sock-reader-#26502%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Connection check done: [liveAddr=/127.0.0.1:47501, previousNode=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], addressesToCheck=[/127.0.0.1:47501], connectingNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:45,511][INFO ][tcp-disco-sock-reader-#26503%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:53582, rmtPort=53582] [2018-11-14 09:49:45,512][INFO ][tcp-disco-sock-reader-#26502%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=true, lastMessageReceivedTime=1542188984699, now=1542188985502, connCheckInterval=500] [2018-11-14 09:49:45,512][WARN ][tcp-disco-sock-reader-#26503%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Failed to read magic header (too few bytes received) [rmtAddr=/127.0.0.1:53582, locAddr=/127.0.0.1:47501] [2018-11-14 09:49:45,513][INFO ][tcp-disco-sock-reader-#26503%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:53582, rmtPort=53582 [2018-11-14 09:49:45,513][INFO ][tcp-disco-sock-reader-#26502%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:57367, rmtPort=57367 [2018-11-14 09:49:45,513][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node has connection to it's previous, trying previous again. [next=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,514][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,514][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=48293] [2018-11-14 09:49:45,515][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=48293] [2018-11-14 09:49:45,515][INFO ][tcp-disco-sock-reader-#26504%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:48293, rmtPort=48293] [2018-11-14 09:49:45,515][INFO ][tcp-disco-sock-reader-#26504%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985512, connCheckInterval=500] [2018-11-14 09:49:45,516][INFO ][tcp-disco-sock-reader-#26504%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:48293, rmtPort=48293 [2018-11-14 09:49:45,716][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node, try previous [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,719][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,720][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=58408] [2018-11-14 09:49:45,720][INFO ][tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=58408] [2018-11-14 09:49:45,720][INFO ][tcp-disco-sock-reader-#26505%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:58408, rmtPort=58408] [2018-11-14 09:49:45,721][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,721][INFO ][tcp-disco-sock-reader-#26505%tcp.TcpDiscoveryMultiThreadedTest5%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:58408, rmtPort=58408 [2018-11-14 09:49:45,724][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,725][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=34401] [2018-11-14 09:49:45,725][INFO ][tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=34401] [2018-11-14 09:49:45,725][INFO ][tcp-disco-sock-reader-#26506%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:34401, rmtPort=34401] [2018-11-14 09:49:45,726][INFO ][tcp-disco-sock-reader-#26506%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=0, now=1542188985723, connCheckInterval=500] [2018-11-14 09:49:45,726][INFO ][tcp-disco-sock-reader-#26506%tcp.TcpDiscoveryMultiThreadedTest4%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:34401, rmtPort=34401 [2018-11-14 09:49:45,726][WARN ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Failed to send message to next node [msg=TcpDiscoveryCustomEventMessage [msg=StartRoutineDiscoveryMessage [startReqData=StartRequestData [prjPred=IgniteAllNodesPredicate [], clsName=o.a.i.configuration.CacheConfiguration$IgniteAllNodesPredicate, depInfo=GridDeploymentInfoBean [clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, depMode=SHARED, userVer=0, locDepOwner=false, participants=null], hnd=CacheContinuousQueryHandler [returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@7c2d9e32, cacheName=default, rmtFilter=null, rmtFilterDep=null, internal=false, notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=false, ackBuf=CacheContinuousQueryAcknowledgeBuffer [size=0, updateCntrs=HashMap {}, topVers=HashSet []], cacheId=1544803905, initTopVer=null, nodeLeft=false, ignoreClsNotFound=false, nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=false, routineId=004d0e16-46be-43c0-a92a-844f8c829aa4], super=TcpDiscoveryAbstractMessage [sndNodeId=ae293efb-8b41-4b86-b32b-747070800001, id=7f213a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, verifierNodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=15, pendingIdx=0, failedNodes=null, isClient=false]], next=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], errMsg=N/A] [2018-11-14 09:49:45,729][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,729][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=47657] [2018-11-14 09:49:45,729][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=47657] [2018-11-14 09:49:45,730][INFO ][tcp-disco-sock-reader-#26507%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:47657, rmtPort=47657] [2018-11-14 09:49:45,730][INFO ][tcp-disco-sock-reader-#26507%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Previous node alive: [alive=false, lastMessageReceivedTime=1542188984699, now=1542188985723, connCheckInterval=500] [2018-11-14 09:49:45,740][INFO ][tcp-disco-msg-worker-#26383%tcp.TcpDiscoveryMultiThreadedTest0%][GridEncryptionManager] Joining node doesn't have encryption data [node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:45,744][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,744][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=16, locNode=3dc77d76, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,744][WARN ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,745][WARN ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,745][INFO ][tcp-disco-msg-worker-#26454%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=0, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,745][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=17, locNode=3dc77d76, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,745][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=16, locNode=99dc07b8, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,746][INFO ][tcp-disco-srvr-#26487%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=46247] [2018-11-14 09:49:45,746][INFO ][tcp-disco-sock-reader-#26465%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:48800, rmtPort=48800 [2018-11-14 09:49:45,747][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:45,747][WARN ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,747][INFO ][tcp-disco-srvr-#26487%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=46247] [2018-11-14 09:49:45,748][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=17, locNode=99dc07b8, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,748][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:45,749][INFO ][tcp-disco-sock-reader-#26508%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:46247, rmtPort=46247] [2018-11-14 09:49:45,757][WARN ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,757][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=16, locNode=ae293efb, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,757][INFO ][tcp-disco-msg-worker-#26485%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188985753, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,758][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_FAILED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:45,758][WARN ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node FAILED: TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,759][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=60953] [2018-11-14 09:49:45,759][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:45,759][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=60953] [2018-11-14 09:49:45,759][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=17, locNode=ae293efb, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,760][INFO ][tcp-disco-sock-reader-#26509%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:60953, rmtPort=60953] [2018-11-14 09:49:45,761][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188984898, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,762][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,762][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Added new node to topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:45,762][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=18, locNode=ae293efb, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,762][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=18, locNode=3dc77d76, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,763][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=18, locNode=99dc07b8, servers=4, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:45,765][INFO ][disco-notifier-worker-#98868%tcp.TcpDiscoveryMultiThreadedTest2%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188985753, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:45,766][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Nodes started on local machine require more than 80% of physical RAM what can lead to significant slowdown due to swapping (please decrease JVM heap size, data region size or checkpoint buffer size) [required=85199MB, available=96649MB] [2018-11-14 09:49:45,767][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_JOINED, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, customEvt=null, allowMerge=true] [2018-11-14 09:49:45,768][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][IgniteCacheDatabaseSharedManager] Configured data regions started successfully [total=3] [2018-11-14 09:49:45,768][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][msg] Components activation performed in 1 ms. [2018-11-14 09:49:45,771][INFO ][sys-#98876%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [2018-11-14 09:49:45,771][INFO ][sys-#98877%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Started cache [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [2018-11-14 09:49:45,771][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Starting caches on local join performed in 3 ms. [2018-11-14 09:49:45,772][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Skipped waiting for partitions release future (local node is joining) [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,773][INFO ][grid-nio-worker-tcp-comm-1-#98271%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45180, rmtAddr=/127.0.0.1:36626] [2018-11-14 09:49:45,773][INFO ][grid-nio-worker-tcp-comm-1-#98850%tcp.TcpDiscoveryMultiThreadedTest2%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:36626, rmtAddr=/127.0.0.1:45180] [2018-11-14 09:49:45,774][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 2 ms. [2018-11-14 09:49:45,774][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], crd=false] [2018-11-14 09:49:45,967][ERROR][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13222, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=]]] class org.apache.ignite.internal.cluster.ClusterTopologyCheckedException: Failed to send message (node left topology): TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3269) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2858) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) [2018-11-14 09:49:45,975][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Finished sending full message [msgTopVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], groups=[CacheGroupContext [grp=default], CacheGroupContext [grp=ignite-sys-cache]], skipped=[77abe0df-a676-4745-b311-5aa4bb000002], latency=1129ms] [2018-11-14 09:49:45,975][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=15, minorTopVer=0], evt=NODE_FAILED, node=5017e3fa-a0f1-4c55-ae20-110197800004] [2018-11-14 09:49:45,976][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_FAILED, evtNode=4ce119dc-4ac4-43db-b3ca-aba085b00005, customEvt=null, allowMerge=true] [2018-11-14 09:49:45,977][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:45,977][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0]]] [2018-11-14 09:49:45,977][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0]]] [2018-11-14 09:49:45,977][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0]]] [2018-11-14 09:49:45,978][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:45,977][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:45,979][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:45,980][INFO ][sys-#98289%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=16, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001, allReceived=false] [2018-11-14 09:49:45,979][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:45,979][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], crd=true] [2018-11-14 09:49:45,980][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=16, minorTopVer=0], node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, allReceived=true] [2018-11-14 09:49:45,980][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], crd=false] [2018-11-14 09:49:45,980][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:45,981][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], crd=false] [2018-11-14 09:49:45,981][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=16, minorTopVer=0]] [2018-11-14 09:49:45,982][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=17, minorTopVer=0], evt=NODE_FAILED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, evtNodeClient=false] [2018-11-14 09:49:45,982][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, evtNodeClient=false] [2018-11-14 09:49:45,983][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Merge server join exchange, message received [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:45,983][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 1 ms. [2018-11-14 09:49:45,983][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,986][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 2 ms. [2018-11-14 09:49:45,986][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:45,987][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:45,987][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions assignment performed in 0 ms. [2018-11-14 09:49:45,988][WARN ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0-1, 4-9, 12, 14-16, 21, 24, 27-28, 30-31, 33-35, 37, 41, 43, 47, 50, 52, 54-55, 60, 64-68, 70, 74-78, 81, 83-85, 91-93, 96, 99, 102, 106, 108, 110-113, 115-116, 118-120, 123-126, 128-129, 134-135, 139, 142-143, 145-149, 154, 159, 163, 167, 171-172, 180-181, 183, 185, 195-197, 199-200, 202-203, 205-208, 212, 214, 216-220, 222, 224, 226-227, 230-232, 237, 240-242, 247, 254-255, 259, 261, 266-269, 271, 274-275, 279-280, 282-284, 289, 291, 295, 299, 302, 304, 309, 311, 317, 320, 323, 325-327, 329, 331, 335, 342, 344, 347, 350-351, 354-355, 358-359, 361-362, 365-369, 374-375, 377-381, 385-388, 390-391, 393-395, 397-400, 403-404, 412, 414-418, 424, 428, 431, 433, 435-437, 441-442, 447-448, 453, 459-462, 465-466, 468, 471-472, 474-479, 481, 484-485, 487, 489, 491, 493-495, 500, 502, 509-511, 513-514, 518, 520, 522-523, 525, 527-528, 532-533, 536-537, 539-541, 543, 546-549, 551-553, 557-558, 560-564, 568, 571-572, 574, 579, 582-584, 586, 591-593, 595-596, 598, 603-610, 614, 616-617, 619, 621-622, 624, 626-630, 632-635, 640, 645, 648-652, 654, 656, 659, 661, 663-665, 667, 673-674, 676-677, 679-681, 687-694, 699, 702-703, 707-709, 714, 721, 723-725, 728-729, 731, 733, 735-736, 738, 740, 743, 745, 749-750, 753, 756, 760-761, 763, 765-771, 775, 778-779, 783-784, 786-787, 790-791, 794-798, 800-801, 803-806, 809, 813, 815, 817, 820, 822, 824, 830-831, 833, 835, 837, 839-841, 843, 845, 851, 853, 855-856, 858-866, 868, 870, 874-877, 879-883, 885, 887-888, 891, 894-895, 897, 899, 901, 903-904, 906, 908-909, 911, 913, 915-919, 921, 923, 929, 932-938, 940-942, 944-945, 947, 951-952, 954, 958, 960-963, 968, 972, 978-983, 987, 990, 992, 996-998, 1000, 1002, 1008-1009, 1014, 1020], plc=IGNORE] [2018-11-14 09:49:45,990][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:45,991][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:45,991][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:45,991][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:45,991][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null] [2018-11-14 09:49:45,992][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,992][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,992][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:45,992][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,993][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=17, minorTopVer=0], evt=NODE_FAILED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, evtNodeClient=false] [2018-11-14 09:49:45,993][INFO ][sys-#98291%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188983352, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], durationFromInit=10] [2018-11-14 09:49:45,993][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, need merge [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,994][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=18, minorTopVer=0], force=false, evt=NODE_FAILED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:45,994][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:45,994][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, evtNodeClient=false] [2018-11-14 09:49:45,995][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=17, minorTopVer=0], evt=NODE_FAILED, evtNode=77abe0df-a676-4745-b311-5aa4bb000002, evtNodeClient=false] [2018-11-14 09:49:45,997][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Merge exchange future on finish [curFut=AffinityTopologyVersion [topVer=16, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, evtNodeClient=false] [2018-11-14 09:49:45,997][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:45,997][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:45,998][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][CacheAffinitySharedManager] Affinity initialization on local join performed in 2 ms. [2018-11-14 09:49:45,998][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 2 ms. [2018-11-14 09:49:45,998][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 5 ms. [2018-11-14 09:49:45,999][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:45,998][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 6 ms. [2018-11-14 09:49:46,001][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null] [2018-11-14 09:49:46,000][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:46,001][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:46,002][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,002][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null] [2018-11-14 09:49:46,002][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=default, localNodeId=b1b3ec2d-cc09-4372-90a8-57d54c800002 [2018-11-14 09:49:46,003][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=b1b3ec2d-cc09-4372-90a8-57d54c800002 [2018-11-14 09:49:46,002][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], err=null] [2018-11-14 09:49:46,003][INFO ][sys-#98878%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985773, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], durationFromInit=232] [2018-11-14 09:49:46,003][WARN ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0-1, 4-9, 12, 14-16, 21, 24, 27-28, 30-31, 33-35, 37, 41, 43, 47, 50, 52, 54-55, 60, 64-68, 70, 74-78, 81, 83-85, 91-93, 96, 99, 102, 106, 108, 110-113, 115-116, 118-120, 123-126, 128-129, 134-135, 139, 142-143, 145-149, 154, 159, 163, 167, 171-172, 180-181, 183, 185, 195-197, 199-200, 202-203, 205-208, 212, 214, 216-220, 222, 224, 226-227, 230-232, 237, 240-242, 247, 254-255, 259, 261, 266-269, 271, 274-275, 279-280, 282-284, 289, 291, 295, 299, 302, 304, 309, 311, 317, 320, 323, 325-327, 329, 331, 335, 342, 344, 347, 350-351, 354-355, 358-359, 361-362, 365-369, 374-375, 377-381, 385-388, 390-391, 393-395, 397-400, 403-404, 412, 414-418, 424, 428, 431, 433, 435-437, 441-442, 447-448, 453, 459-462, 465-466, 468, 471-472, 474-479, 481, 484-485, 487, 489, 491, 493-495, 500, 502, 509-511, 513-514, 518, 520, 522-523, 525, 527-528, 532-533, 536-537, 539-541, 543, 546-549, 551-553, 557-558, 560-564, 568, 571-572, 574, 579, 582-584, 586, 591-593, 595-596, 598, 603-610, 614, 616-617, 619, 621-622, 624, 626-630, 632-635, 640, 645, 648-652, 654, 656, 659, 661, 663-665, 667, 673-674, 676-677, 679-681, 687-694, 699, 702-703, 707-709, 714, 721, 723-725, 728-729, 731, 733, 735-736, 738, 740, 743, 745, 749-750, 753, 756, 760-761, 763, 765-771, 775, 778-779, 783-784, 786-787, 790-791, 794-798, 800-801, 803-806, 809, 813, 815, 817, 820, 822, 824, 830-831, 833, 835, 837, 839-841, 843, 845, 851, 853, 855-856, 858-866, 868, 870, 874-877, 879-883, 885, 887-888, 891, 894-895, 897, 899, 901, 903-904, 906, 908-909, 911, 913, 915-919, 921, 923, 929, 932-938, 940-942, 944-945, 947, 951-952, 954, 958, 960-963, 968, 972, 978-983, 987, 990, 992, 996-998, 1000, 1002, 1008-1009, 1014, 1020], plc=IGNORE] [2018-11-14 09:49:46,004][WARN ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[0-1, 4-9, 12, 14-16, 21, 24, 27-28, 30-31, 33-35, 37, 41, 43, 47, 50, 52, 54-55, 60, 64-68, 70, 74-78, 81, 83-85, 91-93, 96, 99, 102, 106, 108, 110-113, 115-116, 118-120, 123-126, 128-129, 134-135, 139, 142-143, 145-149, 154, 159, 163, 167, 171-172, 180-181, 183, 185, 195-197, 199-200, 202-203, 205-208, 212, 214, 216-220, 222, 224, 226-227, 230-232, 237, 240-242, 247, 254-255, 259, 261, 266-269, 271, 274-275, 279-280, 282-284, 289, 291, 295, 299, 302, 304, 309, 311, 317, 320, 323, 325-327, 329, 331, 335, 342, 344, 347, 350-351, 354-355, 358-359, 361-362, 365-369, 374-375, 377-381, 385-388, 390-391, 393-395, 397-400, 403-404, 412, 414-418, 424, 428, 431, 433, 435-437, 441-442, 447-448, 453, 459-462, 465-466, 468, 471-472, 474-479, 481, 484-485, 487, 489, 491, 493-495, 500, 502, 509-511, 513-514, 518, 520, 522-523, 525, 527-528, 532-533, 536-537, 539-541, 543, 546-549, 551-553, 557-558, 560-564, 568, 571-572, 574, 579, 582-584, 586, 591-593, 595-596, 598, 603-610, 614, 616-617, 619, 621-622, 624, 626-630, 632-635, 640, 645, 648-652, 654, 656, 659, 661, 663-665, 667, 673-674, 676-677, 679-681, 687-694, 699, 702-703, 707-709, 714, 721, 723-725, 728-729, 731, 733, 735-736, 738, 740, 743, 745, 749-750, 753, 756, 760-761, 763, 765-771, 775, 778-779, 783-784, 786-787, 790-791, 794-798, 800-801, 803-806, 809, 813, 815, 817, 820, 822, 824, 830-831, 833, 835, 837, 839-841, 843, 845, 851, 853, 855-856, 858-866, 868, 870, 874-877, 879-883, 885, 887-888, 891, 894-895, 897, 899, 901, 903-904, 906, 908-909, 911, 913, 915-919, 921, 923, 929, 932-938, 940-942, 944-945, 947, 951-952, 954, 958, 960-963, 968, 972, 978-983, 987, 990, 992, 996-998, 1000, 1002, 1008-1009, 1014, 1020], plc=IGNORE] [2018-11-14 09:49:46,004][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][GridCachePartitionExchangeManager] Rebalancing scheduled [order=[ignite-sys-cache, default], top=AffinityTopologyVersion [topVer=18, minorTopVer=0], force=false, evt=NODE_JOINED, node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:46,009][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 6 ms. [2018-11-14 09:49:46,007][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 4 ms. [2018-11-14 09:49:46,009][INFO ][exchange-worker-#98872%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Prepared rebalancing [grp=ignite-sys-cache, mode=SYNC, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, partitionsCount=100, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], parallelism=1] [2018-11-14 09:49:46,010][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=18, minorTopVer=0], force=false, evt=NODE_FAILED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:46,011][INFO ][sys-#98767%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], durationFromInit=262] [2018-11-14 09:49:46,011][ERROR][node-restart-1][root] Unexpected error: javax.cache.CacheException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] javax.cache.CacheException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:376) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:695) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:710) at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:610) at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:226) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.getOrCreatePartitionRecovery(CacheContinuousQueryHandler.java:991) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.waitTopologyFuture(CacheContinuousQueryHandler.java:651) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery0(CacheContinuousQueryManager.java:751) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery(CacheContinuousQueryManager.java:567) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.queryContinuous(IgniteCacheProxyImpl.java:590) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:679) ... 6 more [2018-11-14 09:49:46,010][INFO ][sys-#98881%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Started rebalance routine [ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topic=0, fullPartitions=[0-99], histPartitions=[]] [2018-11-14 09:49:46,010][ERROR][node-restart-3][root] Unexpected error: javax.cache.CacheException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] javax.cache.CacheException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:376) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:695) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:710) at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:610) at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:226) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.getOrCreatePartitionRecovery(CacheContinuousQueryHandler.java:991) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.waitTopologyFuture(CacheContinuousQueryHandler.java:651) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery0(CacheContinuousQueryManager.java:751) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery(CacheContinuousQueryManager.java:567) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.queryContinuous(IgniteCacheProxyImpl.java:590) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:679) ... 6 more [2018-11-14 09:49:46,034][WARN ][node-restart-2][TcpDiscoveryMultiThreadedTest2] Grid startup routine has been interrupted (will rollback). [2018-11-14 09:49:46,033][ERROR][node-restart-5][TcpDiscoveryMultiThreadedTest5] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager] class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@18ea00a2], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false] at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:937) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1683) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1067) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been interrupted. at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:958) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) ... 17 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:951) ... 20 more [2018-11-14 09:49:46,033][ERROR][node-restart-4][TcpDiscoveryMultiThreadedTest4] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager] class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@427919dd], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false] at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:937) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1683) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1067) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been interrupted. at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:958) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) ... 17 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:951) ... 20 more [2018-11-14 09:49:46,012][INFO ][utility-#98882%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed (final) rebalancing [grp=ignite-sys-cache, supplier=3dc77d76-35d6-4baf-b057-c1ecfcf00000, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], progress=1/1, time=0 ms] [2018-11-14 09:49:46,011][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ae293efb-8b41-4b86-b32b-747070800001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=16, minorTopVer=0], evt=NODE_FAILED, evtNode=TcpDiscoveryNode [id=4ce119dc-4ac4-43db-b3ca-aba085b00005, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47504], discPort=47504, order=5, intOrder=5, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], durationFromInit=252] [2018-11-14 09:49:46,011][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=18, minorTopVer=0], force=false, evt=NODE_FAILED, node=4ce119dc-4ac4-43db-b3ca-aba085b00005] [2018-11-14 09:49:46,011][INFO ][utility-#98530%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionSupplier] Finished supplying rebalancing [grp=ignite-sys-cache, demander=b1b3ec2d-cc09-4372-90a8-57d54c800002, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], topic=0] [2018-11-14 09:49:46,051][INFO ][utility-#98882%tcp.TcpDiscoveryMultiThreadedTest2%][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=ignite-sys-cache], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1, routines=1] [2018-11-14 09:49:46,050][WARN ][node-restart-4][TcpDiscoveryMultiThreadedTest4] Grid startup routine has been interrupted (will rollback). [2018-11-14 09:49:46,042][WARN ][node-restart-5][TcpDiscoveryMultiThreadedTest5] Grid startup routine has been interrupted (will rollback). [2018-11-14 09:49:46,038][INFO ][node-restart-2][GridDhtPartitionDemander] Cancelled rebalancing from all nodes [grp=default, topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0]] [2018-11-14 09:49:46,054][INFO ][node-restart-2][GridDhtPartitionDemander] Completed rebalance future: RebalanceFuture [grp=CacheGroupContext [grp=default], topVer=AffinityTopologyVersion [topVer=18, minorTopVer=0], rebalanceId=1, routines=3] [2018-11-14 09:49:46,055][WARN ][grid-nio-worker-tcp-comm-0-#98843%tcp.TcpDiscoveryMultiThreadedTest5%][TcpCommunicationSpi] Local node is not started or fully initialized [isStopping=true] [2018-11-14 09:49:46,056][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,056][INFO ][tcp-disco-sock-reader-#26508%tcp.TcpDiscoveryMultiThreadedTest2%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:46247, rmtPort=46247 [2018-11-14 09:49:46,056][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,057][INFO ][node-restart-5][TcpDiscoverySpi] No verification for local node leave has been received from coordinator (will stop node anyway). [2018-11-14 09:49:46,056][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188984898, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,056][INFO ][tcp-disco-msg-worker-#26454%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,058][INFO ][disco-event-worker-#98283%tcp.TcpDiscoveryMultiThreadedTest0%][GridDiscoveryManager] Topology snapshot [ver=19, locNode=3dc77d76, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,057][WARN ][node-restart-5][TcpDiscoverySpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-disco-srvr, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, finished=false, heartbeatTs=9223372036854775807, hashCode=424539875, interrupted=true, runner=tcp-disco-srvr-#26486%tcp.TcpDiscoveryMultiThreadedTest5%] [2018-11-14 09:49:46,057][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=19, locNode=ae293efb, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,057][INFO ][node-restart-4][TcpDiscoverySpi] No verification for local node leave has been received from coordinator (will stop node anyway). [2018-11-14 09:49:46,057][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:46,060][WARN ][node-restart-4][TcpDiscoverySpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-disco-srvr, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, finished=true, heartbeatTs=1542188986055, hashCode=980901797, interrupted=true, runner=tcp-disco-srvr-#26471%tcp.TcpDiscoveryMultiThreadedTest4%] [2018-11-14 09:49:46,056][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=19, locNode=99dc07b8, servers=3, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,061][WARN ][node-restart-4][TcpDiscoverySpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, finished=true, heartbeatTs=1542188986045, hashCode=605075016, interrupted=unknown, runner=null], nextNode=[null] [2018-11-14 09:49:46,060][INFO ][node-restart-2][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:46,060][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=14, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_LEFT, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,060][WARN ][node-restart-5][TcpDiscoverySpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, finished=true, heartbeatTs=1542188986045, hashCode=978351853, interrupted=unknown, runner=null], nextNode=[null] [2018-11-14 09:49:46,059][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_LEFT, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,059][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=50287] [2018-11-14 09:49:46,063][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,063][WARN ][node-restart-5][SharedFsCheckpointSpi] Got interrupted while waiting for completion of a thread: SharedFsTimeoutTask [marshaller=JdkMarshaller [clsFilter=null], lsnr=o.a.i.i.managers.checkpoint.GridCheckpointManager$1@10db5168] [2018-11-14 09:49:46,063][WARN ][node-restart-4][SharedFsCheckpointSpi] Got interrupted while waiting for completion of a thread: SharedFsTimeoutTask [marshaller=JdkMarshaller [clsFilter=null], lsnr=o.a.i.i.managers.checkpoint.GridCheckpointManager$1@58f5cbfe] [2018-11-14 09:49:46,062][INFO ][node-restart-2][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188985984, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=d8413a11761-b1b3ec2d-cc09-4372-90a8-57d54c800002, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:46,062][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,061][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=3dc77d76-35d6-4baf-b057-c1ecfcf00000, crdVer=1542188983103, topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0]], mvccCrdChange=false, crd=false, evt=NODE_LEFT, evtNode=b1b3ec2d-cc09-4372-90a8-57d54c800002, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,065][WARN ][node-restart-2][SharedFsCheckpointSpi] Got interrupted while waiting for completion of a thread: SharedFsTimeoutTask [marshaller=JdkMarshaller [clsFilter=null], lsnr=o.a.i.i.managers.checkpoint.GridCheckpointManager$1@2b9acfd1] [2018-11-14 09:49:46,064][WARN ][node-restart-4][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=nio-acceptor-tcp-comm, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, finished=true, heartbeatTs=1542188986055, hashCode=786546039, interrupted=true, runner=nio-acceptor-tcp-comm-#98780%tcp.TcpDiscoveryMultiThreadedTest4%] [2018-11-14 09:49:46,064][WARN ][node-restart-5][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=nio-acceptor-tcp-comm, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, finished=true, heartbeatTs=1542188986055, hashCode=1964426031, interrupted=true, runner=nio-acceptor-tcp-comm-#98847%tcp.TcpDiscoveryMultiThreadedTest5%] [2018-11-14 09:49:46,063][INFO ][tcp-disco-srvr-#26384%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=50287] [2018-11-14 09:49:46,067][WARN ][node-restart-5][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: DirectNioClientWorker [super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=18, bytesRcvd0=0, bytesSent0=18, select=false, super=GridWorker [name=grid-nio-worker-tcp-comm-0, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, finished=true, heartbeatTs=1542188986055, hashCode=1645126533, interrupted=true, runner=grid-nio-worker-tcp-comm-0-#98843%tcp.TcpDiscoveryMultiThreadedTest5%]]] [2018-11-14 09:49:46,067][WARN ][node-restart-4][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=false, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, finished=true, heartbeatTs=1542188984686, hashCode=1203125805, interrupted=true, runner=grid-nio-worker-tcp-comm-1-#98777%tcp.TcpDiscoveryMultiThreadedTest4%]]] [2018-11-14 09:49:46,066][WARN ][node-restart-2][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=nio-acceptor-tcp-comm, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, finished=false, heartbeatTs=1542188986065, hashCode=1463655163, interrupted=true, runner=nio-acceptor-tcp-comm-#98853%tcp.TcpDiscoveryMultiThreadedTest2%] [2018-11-14 09:49:46,066][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,069][WARN ][node-restart-2][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: DirectNioClientWorker [super=AbstractNioClientWorker [idx=1, bytesRcvd=26315, bytesSent=1030, bytesRcvd0=26315, bytesSent0=1030, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-1, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, finished=false, heartbeatTs=1542188986055, hashCode=683132869, interrupted=true, runner=grid-nio-worker-tcp-comm-1-#98850%tcp.TcpDiscoveryMultiThreadedTest2%]]] [2018-11-14 09:49:46,068][WARN ][node-restart-4][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-comm-worker, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest4, finished=true, heartbeatTs=1542188986065, hashCode=1511419936, interrupted=unknown, runner=null] [2018-11-14 09:49:46,068][WARN ][node-restart-5][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-comm-worker, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest5, finished=true, heartbeatTs=1542188986065, hashCode=1734205145, interrupted=unknown, runner=null] [2018-11-14 09:49:46,067][INFO ][tcp-disco-sock-reader-#26511%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:50287, rmtPort=50287] [2018-11-14 09:49:46,071][ERROR][node-restart-5][TcpDiscoveryMultiThreadedTest5] Failed to stop component (ignoring): GridProcessorAdapter [] class org.apache.ignite.internal.IgniteInterruptedCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7796) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.stop(GridTimeoutProcessor.java:87) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2356) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2228) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1171) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.ignite.internal.util.worker.GridWorker.join(GridWorker.java:243) at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7791) ... 18 more [2018-11-14 09:49:46,070][ERROR][node-restart-4][TcpDiscoveryMultiThreadedTest4] Failed to stop component (ignoring): GridProcessorAdapter [] class org.apache.ignite.internal.IgniteInterruptedCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7796) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.stop(GridTimeoutProcessor.java:87) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2356) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2228) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1171) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.ignite.internal.util.worker.GridWorker.join(GridWorker.java:243) at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7791) ... 18 more [2018-11-14 09:49:46,070][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]]] [2018-11-14 09:49:46,070][WARN ][node-restart-2][TcpCommunicationSpi] Got interrupted while waiting for completion of runnable: GridWorker [name=tcp-comm-worker, igniteInstanceName=tcp.TcpDiscoveryMultiThreadedTest2, finished=true, heartbeatTs=1542188986065, hashCode=1766932374, interrupted=true, runner=tcp-comm-worker-#26482%tcp.TcpDiscoveryMultiThreadedTest2%] [2018-11-14 09:49:46,069][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]]] [2018-11-14 09:49:46,069][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]]] [2018-11-14 09:49:46,078][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,077][ERROR][node-restart-2][TcpDiscoveryMultiThreadedTest2] Failed to stop component (ignoring): GridProcessorAdapter [] class org.apache.ignite.internal.IgniteInterruptedCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7796) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor.stop(GridTimeoutProcessor.java:87) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2356) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2228) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1171) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.ignite.internal.util.worker.GridWorker.join(GridWorker.java:243) at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:7791) ... 18 more [2018-11-14 09:49:46,077][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,076][INFO ][node-restart-4][TcpDiscoveryMultiThreadedTest4] >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped with ERRORS >>> +---------------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest4 >>> Grid uptime: 00:00:01.439 >>> See log above for detailed error message. >>> Note that some errors during stop can prevent grid from >>> maintaining correct topology since this node may have >>> not exited grid properly. [2018-11-14 09:49:46,074][INFO ][node-restart-5][TcpDiscoveryMultiThreadedTest5] >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped with ERRORS >>> +---------------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest5 >>> Grid uptime: 00:00:01.296 >>> See log above for detailed error message. >>> Note that some errors during stop can prevent grid from >>> maintaining correct topology since this node may have >>> not exited grid properly. [2018-11-14 09:49:46,082][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:46,081][INFO ][node-restart-2][TcpDiscoveryMultiThreadedTest2] >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped with ERRORS >>> +---------------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest2 >>> Grid uptime: 00:00:01.269 >>> See log above for detailed error message. >>> Note that some errors during stop can prevent grid from >>> maintaining correct topology since this node may have >>> not exited grid properly. [2018-11-14 09:49:46,079][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 1 ms. [2018-11-14 09:49:46,084][ERROR][node-restart-4][root] Unexpected error: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1688) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1067) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@427919dd], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false] at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:937) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1683) ... 15 more Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been interrupted. at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:958) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) ... 17 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:951) ... 20 more [2018-11-14 09:49:46,078][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,089][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=19, minorTopVer=0], node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, allReceived=false] [2018-11-14 09:49:46,089][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=19, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001, allReceived=true] [2018-11-14 09:49:46,090][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=19, minorTopVer=0]] [2018-11-14 09:49:46,086][ERROR][node-restart-2][root] Unexpected error: class org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted while waiting for future to complete. class org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:186) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:1024) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1125) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) [2018-11-14 09:49:46,086][ERROR][node-restart-5][root] Unexpected error: class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite.internal.managers.discovery.GridDiscoveryManager] at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1688) at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1067) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732) at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158) at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:656) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:935) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:876) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:864) at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:830) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest.access$2600(TcpDiscoveryMultiThreadedTest.java:71) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:680) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@18ea00a2], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false] at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:300) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:937) at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1683) ... 15 more Caused by: class org.apache.ignite.spi.IgniteSpiException: Thread has been interrupted. at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:958) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2020) at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297) ... 17 more Caused by: java.lang.InterruptedException at java.lang.Object.wait(Native Method) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:951) ... 20 more [2018-11-14 09:49:46,084][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], crd=false] [2018-11-14 09:49:46,083][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], crd=false] [2018-11-14 09:49:46,090][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:46,106][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]] [2018-11-14 09:49:46,107][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:46,108][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 2 ms. [2018-11-14 09:49:46,108][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:46,108][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,109][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 0 ms. [2018-11-14 09:49:46,110][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:46,110][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:46,110][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], err=null] [2018-11-14 09:49:46,110][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]] [2018-11-14 09:49:46,111][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:46,111][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=3dc77d76-35d6-4baf-b057-c1ecfcf00000, resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0]] [2018-11-14 09:49:46,112][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 1 ms. [2018-11-14 09:49:46,112][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 0 ms. [2018-11-14 09:49:46,113][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], err=null] [2018-11-14 09:49:46,113][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,114][INFO ][sys-#98765%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ae293efb-8b41-4b86-b32b-747070800001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], durationFromInit=50] [2018-11-14 09:49:46,110][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,115][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188984898, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], durationFromInit=50] [2018-11-14 09:49:46,114][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=19, minorTopVer=0], force=false, evt=NODE_LEFT, node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:46,112][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity applying from full message performed in 0 ms. [2018-11-14 09:49:46,117][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], crd=true] [2018-11-14 09:49:46,117][ERROR][main][root] Test failed. class org.apache.ignite.IgniteException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:708) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:676) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:816) at org.apache.ignite.testframework.GridTestUtils$3.call(GridTestUtils.java:814) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: javax.cache.CacheException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:697) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:376) at org.apache.ignite.spi.discovery.tcp.TcpDiscoveryMultiThreadedTest$9.apply(TcpDiscoveryMultiThreadedTest.java:695) ... 4 more Caused by: java.lang.IllegalStateException: Getting affinity for topology version earlier than affinity is calculated [locNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188985763, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], grp=default, topVer=AffinityTopologyVersion [topVer=17, minorTopVer=0], head=AffinityTopologyVersion [topVer=18, minorTopVer=0], history=[AffinityTopologyVersion [topVer=14, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0], AffinityTopologyVersion [topVer=18, minorTopVer=0]]] at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.cachedAffinity(GridAffinityAssignmentCache.java:710) at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:610) at org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodesByPartition(GridCacheAffinityManager.java:226) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.getOrCreatePartitionRecovery(CacheContinuousQueryHandler.java:991) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryHandler.waitTopologyFuture(CacheContinuousQueryHandler.java:651) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery0(CacheContinuousQueryManager.java:751) at org.apache.ignite.internal.processors.cache.query.continuous.CacheContinuousQueryManager.executeQuery(CacheContinuousQueryManager.java:567) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.queryContinuous(IgniteCacheProxyImpl.java:590) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:679) ... 6 more [2018-11-14 09:49:46,118][INFO ][exchange-worker-#98284%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=19, minorTopVer=0], force=false, evt=NODE_LEFT, node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:46,118][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 6 ms. [2018-11-14 09:49:46,123][INFO ][main][root] >>> Stopping test: TcpDiscoveryMultiThreadedTest#testCustomEventNodeRestart in 3053 ms <<< [2018-11-14 09:49:46,124][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:46,124][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], err=null] [2018-11-14 09:49:46,124][INFO ][main][root] >>> Stopping grid [name=tcp.TcpDiscoveryMultiThreadedTest0, id=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:46,124][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,125][INFO ][sys-#98762%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=b1b3ec2d-cc09-4372-90a8-57d54c800002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47505], discPort=47505, order=18, intOrder=11, lastExchangeTime=1542188985743, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=19, minorTopVer=0], durationFromInit=61] [2018-11-14 09:49:46,125][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=19, minorTopVer=0], force=false, evt=NODE_LEFT, node=b1b3ec2d-cc09-4372-90a8-57d54c800002] [2018-11-14 09:49:46,126][INFO ][disco-notifier-worker-#98730%tcp.TcpDiscoveryMultiThreadedTest3%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188986126, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,127][INFO ][tcp-disco-sock-reader-#26511%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:50287, rmtPort=50287 [2018-11-14 09:49:46,127][INFO ][tcp-disco-msg-worker-#26454%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,127][INFO ][tcp-disco-sock-reader-#26509%tcp.TcpDiscoveryMultiThreadedTest0%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:60953, rmtPort=60953 [2018-11-14 09:49:46,127][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,127][INFO ][disco-notifier-worker-#98729%tcp.TcpDiscoveryMultiThreadedTest1%][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]], crdNode=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,129][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=20, locNode=99dc07b8, servers=2, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,128][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=46678] [2018-11-14 09:49:46,130][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Coordinator changed [prev=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], cur=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188986126, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,130][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:46,129][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,131][INFO ][sys-#98737%tcp.TcpDiscoveryMultiThreadedTest3%][ExchangeLatchManager] Become new coordinator 99dc07b8-e8a8-49ca-bcf9-392b48e00003 [2018-11-14 09:49:46,131][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]], mvccCrdChange=true, crd=true, evt=NODE_LEFT, evtNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,131][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:46,130][INFO ][tcp-disco-srvr-#26456%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=46678] [2018-11-14 09:49:46,132][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Topology snapshot [ver=20, locNode=ae293efb, servers=2, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,133][INFO ][disco-event-worker-#98738%tcp.TcpDiscoveryMultiThreadedTest1%][GridDiscoveryManager] Coordinator changed [prev=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], cur=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false]] [2018-11-14 09:49:46,134][INFO ][tcp-disco-sock-reader-#26512%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:46678, rmtPort=46678] [2018-11-14 09:49:46,135][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]], mvccCrdChange=true, crd=false, evt=NODE_LEFT, evtNode=3dc77d76-35d6-4baf-b057-c1ecfcf00000, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,136][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,136][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,136][INFO ][grid-nio-worker-tcp-comm-3-#98710%tcp.TcpDiscoveryMultiThreadedTest3%][TcpCommunicationSpi] Accepted incoming communication connection [locAddr=/127.0.0.1:45182, rmtAddr=/127.0.0.1:59878] [2018-11-14 09:49:46,137][INFO ][grid-nio-worker-tcp-comm-3-#98706%tcp.TcpDiscoveryMultiThreadedTest1%][TcpCommunicationSpi] Established outgoing communication connection [locAddr=/127.0.0.1:59878, rmtAddr=/127.0.0.1:45182] [2018-11-14 09:49:46,138][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]]] [2018-11-14 09:49:46,138][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ClientLatch [coordinator=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], ackSent=true, super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]]] [2018-11-14 09:49:46,138][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,139][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], crd=true] [2018-11-14 09:49:46,139][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,140][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Sending Single Message performed in 0 ms. [2018-11-14 09:49:46,140][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], crd=false] [2018-11-14 09:49:46,141][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Coordinator received single message [ver=AffinityTopologyVersion [topVer=20, minorTopVer=0], node=ae293efb-8b41-4b86-b32b-747070800001, allReceived=true] [2018-11-14 09:49:46,141][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=20, minorTopVer=0]] [2018-11-14 09:49:46,142][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:46,142][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]] [2018-11-14 09:49:46,143][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:46,145][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 3 ms. [2018-11-14 09:49:46,145][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Partitions validation performed in 0 ms. [2018-11-14 09:49:46,146][WARN ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[1-3, 9-11, 13, 19-21, 25, 32, 38, 50, 52-53, 69, 72, 78-79, 81-82, 87, 89, 97, 100, 108-109, 116, 121-122, 127, 130, 132, 134, 137, 148, 150-152, 157-158, 160, 164-166, 168, 175-176, 181-182, 186, 188-189, 191, 198-199, 205, 210-211, 225-226, 228-229, 234-236, 239, 243, 245, 248, 251, 260, 263, 266-269, 273, 275, 281-282, 285-286, 301, 304-307, 312-313, 316, 324, 334, 336, 341, 343, 345, 349, 353, 358, 360, 363-364, 366, 370-372, 375-376, 384, 390, 394-396, 399, 405, 410-412, 416, 421, 426-427, 429, 433-434, 437-440, 445-446, 448-452, 455-457, 461-462, 467, 470-471, 475-476, 482, 486, 495, 499-501, 504-505, 507-508, 511, 513-514, 519, 522, 524, 529-530, 535, 544, 546, 551, 556-557, 562, 567, 571, 573, 575, 577, 589-590, 594, 600, 605-606, 609, 613, 615, 623, 625, 631, 633, 636, 638, 641-642, 646-647, 649, 652-653, 659, 664, 666-667, 669-670, 678, 683-685, 690, 692, 694, 697-698, 700-701, 709, 711, 715, 719, 721, 726-727, 729-730, 732, 735, 739, 753, 755, 758, 762-765, 770, 774, 779, 781-782, 785, 789-790, 792, 799-801, 810-811, 816, 818, 821, 825-826, 831, 836-839, 842, 844, 850, 852, 855, 857, 865, 867, 871, 874, 878, 884, 886, 888, 893-894, 897-898, 900, 902, 904, 907, 910, 917, 926, 928, 934, 939, 942, 945, 948, 950, 956, 958, 963, 965, 967, 971, 973-974, 976-977, 981, 985, 988-989, 992, 999-1001, 1011, 1015-1016, 1018-1019, 1021-1023], plc=IGNORE] [2018-11-14 09:49:46,148][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:46,149][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Preparing Full Message performed in 1 ms. [2018-11-14 09:49:46,149][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Full Message performed in 0 ms. [2018-11-14 09:49:46,149][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Sending Full Message to all nodes performed in 0 ms. [2018-11-14 09:49:46,150][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], err=null] [2018-11-14 09:49:46,150][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Received full message, will finish exchange [node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]] [2018-11-14 09:49:46,150][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 0 ms. [2018-11-14 09:49:46,151][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][MvccProcessorImpl] Initialize local node as mvcc coordinator [node=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115] [2018-11-14 09:49:46,151][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][CacheAffinitySharedManager] Affinity applying from full message performed in 1 ms. [2018-11-14 09:49:46,152][INFO ][sys-#98766%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], durationFromInit=20] [2018-11-14 09:49:46,153][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=20, minorTopVer=0], force=false, evt=NODE_LEFT, node=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:46,154][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Affinity changes applied in 4 ms. [2018-11-14 09:49:46,155][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Full map updating for 2 groups performed in 1 ms. [2018-11-14 09:49:46,155][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], err=null] [2018-11-14 09:49:46,155][WARN ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionTopologyImpl] Detected lost partitions [grp=default, parts=[1-3, 9-11, 13, 19-21, 25, 32, 38, 50, 52-53, 69, 72, 78-79, 81-82, 87, 89, 97, 100, 108-109, 116, 121-122, 127, 130, 132, 134, 137, 148, 150-152, 157-158, 160, 164-166, 168, 175-176, 181-182, 186, 188-189, 191, 198-199, 205, 210-211, 225-226, 228-229, 234-236, 239, 243, 245, 248, 251, 260, 263, 266-269, 273, 275, 281-282, 285-286, 301, 304-307, 312-313, 316, 324, 334, 336, 341, 343, 345, 349, 353, 358, 360, 363-364, 366, 370-372, 375-376, 384, 390, 394-396, 399, 405, 410-412, 416, 421, 426-427, 429, 433-434, 437-440, 445-446, 448-452, 455-457, 461-462, 467, 470-471, 475-476, 482, 486, 495, 499-501, 504-505, 507-508, 511, 513-514, 519, 522, 524, 529-530, 535, 544, 546, 551, 556-557, 562, 567, 571, 573, 575, 577, 589-590, 594, 600, 605-606, 609, 613, 615, 623, 625, 631, 633, 636, 638, 641-642, 646-647, 649, 652-653, 659, 664, 666-667, 669-670, 678, 683-685, 690, 692, 694, 697-698, 700-701, 709, 711, 715, 719, 721, 726-727, 729-730, 732, 735, 739, 753, 755, 758, 762-765, 770, 774, 779, 781-782, 785, 789-790, 792, 799-801, 810-811, 816, 818, 821, 825-826, 831, 836-839, 842, 844, 850, 852, 855, 857, 865, 867, 871, 874, 878, 884, 886, 888, 893-894, 897-898, 900, 902, 904, 907, 910, 917, 926, 928, 934, 939, 942, 945, 948, 950, 956, 958, 963, 965, 967, 971, 973-974, 976-977, 981, 985, 988-989, 992, 999-1001, 1011, 1015-1016, 1018-1019, 1021-1023], plc=IGNORE] [2018-11-14 09:49:46,157][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Detecting lost partitions performed in 2 ms. [2018-11-14 09:49:46,158][INFO ][sys-#98741%tcp.TcpDiscoveryMultiThreadedTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=ae293efb-8b41-4b86-b32b-747070800001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=3dc77d76-35d6-4baf-b057-c1ecfcf00000, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1542188984481, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0], durationFromInit=30] [2018-11-14 09:49:46,158][INFO ][exchange-worker-#98739%tcp.TcpDiscoveryMultiThreadedTest1%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=20, minorTopVer=0], force=false, evt=NODE_LEFT, node=3dc77d76-35d6-4baf-b057-c1ecfcf00000] [2018-11-14 09:49:46,256][ERROR][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=77abe0df-a676-4745-b311-5aa4bb000002, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=8, intOrder=7, lastExchangeTime=1542188983687, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13223, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=]]] class org.apache.ignite.IgniteCheckedException: Node is stopping. at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7427) at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:172) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2910) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ResendTimeoutObject$1.run(GridCachePartitionExchangeManager.java:3046) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6872) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.spi.IgniteSpiException: Node is stopping. at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3240) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2986) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2869) ... 15 more [2018-11-14 09:49:46,265][ERROR][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984451, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13223, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=]]] class org.apache.ignite.spi.IgniteSpiException: Node is stopping. at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2852) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ResendTimeoutObject$1.run(GridCachePartitionExchangeManager.java:3046) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6872) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2018-11-14 09:49:46,272][ERROR][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][TcpCommunicationSpi] Failed to send message to remote node [node=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188984431, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridCacheMessage [msgId=13223, depInfo=null, err=null, skipPrepare=false]GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null, super=]GridDhtPartitionsFullMessage [parts=HashMap {1544803905=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=43, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=581, size=288], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=1493, size=206], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=70, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=142, size=208], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=26, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=627, size=278], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=129, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=97, size=222]}, -2100569601=GridDhtPartitionFullMap {77abe0df-a676-4745-b311-5aa4bb000002=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=116, size=100], 3dc77d76-35d6-4baf-b057-c1ecfcf00000=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=65, size=100], 99dc07b8-e8a8-49ca-bcf9-392b48e00003=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=15, minorTopVer=0], updateSeq=104, size=100], 4ce119dc-4ac4-43db-b3ca-aba085b00005=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=122, size=100], ae293efb-8b41-4b86-b32b-747070800001=GridDhtPartitionMap [moving=0, top=AffinityTopologyVersion [topVer=14, minorTopVer=0], updateSeq=104, size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null, partsToReload=null, partsSizes=HashMap {1544803905=UnmodifiableMap {0=null, 1=null, 2=0, 3=0, 4=null, 5=null, 6=null, 7=null, 8=null, 9=null, 10=0, 11=0, 12=null, 13=0, 14=null, 15=null, 16=null, 17=null, 18=null, 19=null, 20=0, 21=null, 22=null, 23=null, 24=null, 25=0, 26=null, 27=null, 28=null, 29=null, 30=null, 31=null, 32=null, 33=null, 34=null, 35=null, 36=null, 37=null, 38=0, 39=null, 40=null, 41=null, 42=null, 43=null, 44=0, 45=null, 46=null, 47=null, 48=0, 49=null, 50=null, 51=0, 52=null, 53=0, 54=null, 55=null, 56=null, 57=null, 58=null, 59=null, 60=null, 61=null, 62=null, 63=null, 64=null, 65=null, 66=null, 67=null, 68=null, 69=null, 70=null, 71=null, 72=0, 73=null, 74=null, 75=null, 76=null, 77=null, 78=null, 79=null, 80=null, 81=null, 82=0, 83=null, 84=null, 85=null, 86=null, 87=0, 88=null, 89=0, 90=null, 91=null, 92=null, 93=null, 94=null, 95=0, 96=null, 97=0, 98=0, 99=null... and 924 more}, -2100569601=UnmodifiableMap {0=0, 1=0, 2=0, 3=0, 4=0, 5=0, 6=0, 7=0, 8=0, 9=0, 10=0, 11=0, 12=0, 13=0, 14=0, 15=0, 16=0, 17=0, 18=0, 19=0, 20=0, 21=0, 22=0, 23=0, 24=0, 25=0, 26=0, 27=0, 28=0, 29=0, 30=0, 31=0, 32=0, 33=0, 34=0, 35=0, 36=0, 37=0, 38=0, 39=0, 40=0, 41=0, 42=0, 43=0, 44=0, 45=0, 46=0, 47=0, 48=0, 49=0, 50=0, 51=0, 52=0, 53=0, 54=0, 55=0, 56=0, 57=0, 58=0, 59=0, 60=0, 61=0, 62=0, 63=0, 64=0, 65=0, 66=0, 67=0, 68=0, 69=0, 70=0, 71=0, 72=0, 73=0, 74=0, 75=0, 76=0, 77=0, 78=0, 79=0, 80=0, 81=0, 82=0, 83=0, 84=0, 85=0, 86=0, 87=0, 88=0, 89=0, 90=0, 91=0, 92=0, 93=0, 94=0, 95=0, 96=0, 97=0, 98=0, 99=0}}, topVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], errs=null, compress=true, resTopVer=null, partCnt=2, super=]]] class org.apache.ignite.spi.IgniteSpiException: Node is stopping. at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2852) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2712) at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2671) at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1655) at org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:1730) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1279) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1153) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1098) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1114) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ResendTimeoutObject$1.run(GridCachePartitionExchangeManager.java:3046) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6872) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2018-11-14 09:49:46,279][INFO ][sys-#98288%tcp.TcpDiscoveryMultiThreadedTest0%][GridCachePartitionExchangeManager] Finished sending full message [msgTopVer=AffinityTopologyVersion [topVer=15, minorTopVer=0], groups=[], latency=1300ms] [2018-11-14 09:49:46,281][INFO ][main][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188983122, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=ae5f2a11761-3dc77d76-35d6-4baf-b057-c1ecfcf00000, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.service.GridServiceProcessor$ServiceDeploymentPredicate, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:46,282][INFO ][main][TcpDiscoveryMultiThreadedTest0] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest0 >>> Grid uptime: 00:00:03.165 [2018-11-14 09:49:46,284][INFO ][main][root] >>> Stopping grid [name=tcp.TcpDiscoveryMultiThreadedTest1, id=ae293efb-8b41-4b86-b32b-747070800001] [2018-11-14 09:49:46,287][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false] [2018-11-14 09:49:46,287][INFO ][tcp-disco-sock-reader-#26463%tcp.TcpDiscoveryMultiThreadedTest1%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:56340, rmtPort=56340 [2018-11-14 09:49:46,287][INFO ][disco-event-worker-#98731%tcp.TcpDiscoveryMultiThreadedTest3%][GridDiscoveryManager] Topology snapshot [ver=21, locNode=99dc07b8, servers=1, clients=0, state=ACTIVE, CPUs=5, offheap=19.0GB, heap=2.0GB] [2018-11-14 09:49:46,288][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:46,288][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:46,288][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], mvccCrd=MvccCoordinator [nodeId=99dc07b8-e8a8-49ca-bcf9-392b48e00003, crdVer=1542188983115, topVer=AffinityTopologyVersion [topVer=20, minorTopVer=0]], mvccCrdChange=false, crd=true, evt=NODE_LEFT, evtNode=ae293efb-8b41-4b86-b32b-747070800001, customEvt=null, allowMerge=true] [2018-11-14 09:49:46,289][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2018-11-14 09:49:46,290][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=exchange, topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0]]] [2018-11-14 09:49:46,290][INFO ][main][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188984656, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=8d113a11761-ae293efb-8b41-4b86-b32b-747070800001, userVer=0, loc=true, sampleClsName=org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:46,290][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2018-11-14 09:49:46,291][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=21, minorTopVer=0]] [2018-11-14 09:49:46,291][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Exchanges merging performed in 0 ms. [2018-11-14 09:49:46,292][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=21, minorTopVer=0]] [2018-11-14 09:49:46,291][INFO ][main][TcpDiscoveryMultiThreadedTest1] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest1 >>> Grid uptime: 00:00:01.568 [2018-11-14 09:49:46,293][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][CacheAffinitySharedManager] Affinity recalculation (on server left) performed in 1 ms. [2018-11-14 09:49:46,294][INFO ][main][root] >>> Stopping grid [name=tcp.TcpDiscoveryMultiThreadedTest3, id=99dc07b8-e8a8-49ca-bcf9-392b48e00003] [2018-11-14 09:49:46,298][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Affinity changes (coordinator) applied in 6 ms. [2018-11-14 09:49:46,299][ERROR][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=21, nodeId8=99dc07b8, msg=Node left: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_LEFT, tstamp=1542188986287], nodeId=ae293efb, evt=NODE_LEFT] class org.apache.ignite.IgniteException: Failed to validate partitions state at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.validatePartitionsState(GridDhtPartitionsExchangeFuture.java:3509) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3259) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3157) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1464) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:813) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2759) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.internal.IgniteInterruptedCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10831) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10764) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.validatePartitionsState(GridDhtPartitionsExchangeFuture.java:3474) ... 8 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10826) ... 10 more [2018-11-14 09:49:46,303][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], resVer=null, err=class org.apache.ignite.IgniteException: Failed to validate partitions state] [2018-11-14 09:49:46,303][INFO ][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=99dc07b8-e8a8-49ca-bcf9-392b48e00003, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], done=true], topVer=null, durationFromInit=10] [2018-11-14 09:49:46,304][ERROR][exchange-worker-#98732%tcp.TcpDiscoveryMultiThreadedTest3%][GridCachePartitionExchangeManager] Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=21, nodeId8=99dc07b8, msg=Node left: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_LEFT, tstamp=1542188986287], crd=TcpDiscoveryNode [id=99dc07b8-e8a8-49ca-bcf9-392b48e00003, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47503], discPort=47503, order=13, intOrder=9, lastExchangeTime=1542188986277, loc=true, ver=2.7.0#20181113-sha1:b186327e, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=21, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], topVer=21, nodeId8=99dc07b8, msg=Node left: TcpDiscoveryNode [id=ae293efb-8b41-4b86-b32b-747070800001, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=14, intOrder=10, lastExchangeTime=1542188984461, loc=false, ver=2.7.0#20181113-sha1:b186327e, isClient=false], type=NODE_LEFT, tstamp=1542188986287], nodeId=ae293efb, evt=NODE_LEFT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=2054950547], init=false, lastVer=null, partReleaseFut=AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]AffinityTopologyVersion [topVer=21, minorTopVer=0]PartitionReleaseFuture [topVer=, futures=[ExplicitLockReleaseFuture [topVer=, futures=[]], AtomicUpdateReleaseFuture [topVer=, futures=[]], DataStreamerReleaseFuture [topVer=, futures=[]], LocalTxReleaseFuture [topVer=, futures=[]], AllTxReleaseFuture [topVer=, futures=[RemoteTxReleaseFuture [topVer=, futures=[]]]]]], exchActions=null, affChangeMsg=null, initTs=1542188986287, centralizedAff=false, forceAffReassignment=false, exchangeLocE=null, cacheChangeFailureMsgSent=false, done=true, state=CRD, registerCachesFuture=null, partitionsSent=false, partitionsReceived=false, delayedLatestMsg=null, afterLsnrCompleteFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1891895231], evtLatch=0, remaining=HashSet [], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteException: Failed to validate partitions state, hash=1375221985]] class org.apache.ignite.IgniteCheckedException: Failed to validate partitions state at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7427) at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2787) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2630) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Caused by: class org.apache.ignite.IgniteException: Failed to validate partitions state at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.validatePartitionsState(GridDhtPartitionsExchangeFuture.java:3509) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3259) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3157) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1464) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:813) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2759) ... 3 more Caused by: class org.apache.ignite.internal.IgniteInterruptedCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10831) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10764) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.validatePartitionsState(GridDhtPartitionsExchangeFuture.java:3474) ... 8 more Caused by: java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:10826) ... 10 more [2018-11-14 09:49:46,313][INFO ][tcp-disco-sock-reader-#26507%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:47657, rmtPort=47657 [2018-11-14 09:49:46,313][INFO ][tcp-disco-sock-reader-#26512%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:46678, rmtPort=46678 [2018-11-14 09:49:46,313][INFO ][tcp-disco-sock-reader-#26461%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:39424, rmtPort=39424 [2018-11-14 09:49:46,313][INFO ][tcp-disco-sock-reader-#26467%tcp.TcpDiscoveryMultiThreadedTest3%][TcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:36339, rmtPort=36339 [2018-11-14 09:49:46,314][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2018-11-14 09:49:46,314][INFO ][main][GridCacheProcessor] Stopped cache [cacheName=default] [2018-11-14 09:49:46,317][INFO ][main][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1542188984624, depMode=SHARED, clsLdr=IsolatedClassLoader{roleName='test'}, clsLdrId=ab113a11761-99dc07b8-e8a8-49ca-bcf9-392b48e00003, userVer=0, loc=true, sampleClsName=org.apache.ignite.configuration.CacheConfiguration$IgniteAllNodesPredicate, pendingUndeploy=false, undeployed=true, usage=0] [2018-11-14 09:49:46,318][INFO ][main][TcpDiscoveryMultiThreadedTest3] >>> +------------------------------------------------------------------------------------------+ >>> Ignite ver. 2.7.0-SNAPSHOT#20181113-sha1:b186327eed79f28ce0d5f49cb6f1ee978c89de94 stopped OK >>> +------------------------------------------------------------------------------------------+ >>> Ignite instance name: tcp.TcpDiscoveryMultiThreadedTest3 >>> Grid uptime: 00:00:01.618