WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.ignite.internal.util.GridUnsafe$2 (file:/usr/share/apache-ignite/libs/ignite-core-2.8.1.jar) to field java.nio.Buffer.address WARNING: Please consider reporting this to the maintainers of org.apache.ignite.internal.util.GridUnsafe$2 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Ignite Command Line Startup, ver. 2.8.1#20200521-sha1:86422096 2020 Copyright(C) Apache Software Foundation [12:09:49,421][WARNING][main][G] Ignite work directory is not provided, automatically resolved to: /usr/share/apache-ignite/work [12:09:49,430][WARNING][main][G] Consistent ID is not set, it is recommended to set consistent ID for production clusters (use IgniteConfiguration.setConsistentId property) [12:09:49,589][INFO][main][IgniteKernal] >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.8.1#20200521-sha1:86422096 >>> 2020 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [12:09:49,589][INFO][main][IgniteKernal] Config URL: file:/usr/share/apache-ignite/config/default-config.xml [12:09:49,610][INFO][main][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=24, svcPoolSize=24, callbackPoolSize=24, stripedPoolSize=24, sysPoolSize=24, mgmtPoolSize=4, igfsPoolSize=24, dataStreamerPoolSize=24, utilityCachePoolSize=24, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=24, sqlQryHistSize=1000, dfltQryTimeout=0, igniteHome=/usr/share/apache-ignite, igniteWorkDir=/usr/share/apache-ignite/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@1963006a, nodeId=29a661dd-7405-420d-a230-3250142f10c1, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@301ec38b, chConnPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=12, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@17a1e4ca[Count = 1], stopping=false, metricsLsnr=null], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@10ded6a9, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@56113384, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@5669c5fb, clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=10000, rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, rebalanceBatchSize=524288, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=24, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=0, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=6743085875, initSize=268435456, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0, lazyMemoryAllocation=true], dataRegions=null, storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=db/wal, walArchivePath=db/wal, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@6b3e12b5, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=true, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null, walPageCompression=DISABLED, walPageCompressionLevel=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=24, idleTimeout=0, handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration [maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=true, failureHnd=null, commFailureRslvr=null] [12:09:49,610][INFO][main][IgniteKernal] Daemon mode: off [12:09:49,610][INFO][main][IgniteKernal] OS: Linux 4.15.0-109-generic amd64 [12:09:49,610][INFO][main][IgniteKernal] OS user: root [12:09:49,615][INFO][main][IgniteKernal] PID: 17869 [12:09:49,616][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 11 [12:09:49,616][INFO][main][IgniteKernal] VM information: OpenJDK Runtime Environment 11.0.7+10-post-Ubuntu-2ubuntu218.04 Ubuntu OpenJDK 64-Bit Server VM 11.0.7+10-post-Ubuntu-2ubuntu218.04 [12:09:49,616][INFO][main][IgniteKernal] VM total memory: 20.0GB [12:09:49,617][INFO][main][IgniteKernal] Remote Management [restart: on, REST: on, JMX (remote: on, port: 50278, auth: off, ssl: off)] [12:09:49,617][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=false, config=null] [12:09:49,617][INFO][main][IgniteKernal] IGNITE_HOME=/usr/share/apache-ignite [12:09:49,618][INFO][main][IgniteKernal] VM arguments: [--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED, --add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED, --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED, --illegal-access=permit, -Xms4g, -Xmx20g, -XX:MaxMetaspaceSize=1g, -XX:+UseG1GC, -Xss16m, -Djava.net.preferIPv4Stack=true, -DIGNITE_QUIET=false, -DIGNITE_SUCCESS_FILE=/usr/share/apache-ignite/work/ignite_success_fb4d48e9-a22c-49f8-8bac-d8b7835bc1f2, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=50278, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -DIGNITE_HOME=/usr/share/apache-ignite, -DIGNITE_PROG_NAME=./bin/ignite.sh] [12:09:49,618][INFO][main][IgniteKernal] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [12:09:49,618][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] [12:09:49,622][INFO][main][IgniteKernal] 3-rd party licenses can be found at: /usr/share/apache-ignite/libs/licenses [12:09:49,699][INFO][main][IgnitePluginProcessor] Configured plugins: [12:09:49,699][INFO][main][IgnitePluginProcessor] ^-- Ignite Native I/O Plugin [Direct I/O] [12:09:49,699][INFO][main][IgnitePluginProcessor] ^-- Copyright(C) Apache Software Foundation [12:09:49,700][INFO][main][IgnitePluginProcessor] [12:09:49,703][INFO][main][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] [12:09:49,948][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=12, selectorSpins=0, pairedConn=false] [12:09:49,951][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [12:09:49,970][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [12:09:49,995][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [12:09:49,999][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=off] [12:09:50,041][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=48550, localHost=localhost/127.0.0.1, locNodeId=29a661dd-7405-420d-a230-3250142f10c1] [12:09:50,055][INFO][main][PdsFoldersResolver] Successfully locked persistence storage folder [/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d] [12:09:50,056][INFO][main][PdsFoldersResolver] Consistent ID used for local node is [e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d] according to persistence data storage folders [12:09:50,056][INFO][main][CacheObjectBinaryProcessorImpl] Resolved directory for serialized binary metadata: /usr/share/apache-ignite/work/binary_meta/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d [12:09:50,247][INFO][main][FilePageStoreManager] Resolved page store work directory: /usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d [12:09:50,257][INFO][main][FileWriteAheadLogManager] Resolved write ahead log work directory: /usr/share/apache-ignite/work/db/wal/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d [12:09:50,257][INFO][main][FileWriteAheadLogManager] Resolved write ahead log archive directory: /usr/share/apache-ignite/work/db/wal/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d [12:09:50,281][INFO][main][FileHandleManagerImpl] Initialized write-ahead log manager [mode=LOG_ONLY] [12:09:50,300][WARNING][main][GridCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files [12:09:50,321][INFO][main][GridCacheDatabaseSharedManager] Configured data regions initialized successfully [total=4] [12:09:50,338][INFO][main][PartitionsEvictManager] Evict partition permits=6 [12:09:50,367][WARNING][main][IgniteH2Indexing] Serialization of Java objects in H2 was enabled. [12:09:50,541][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [12:09:50,597][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211] [12:09:50,689][INFO][main][] Page size configuration for storage path [/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d]: 4096; Linux memory page size: 4096; Selected FS block size : 4096. [12:09:50,690][INFO][main][] Selected FS block size : 4096 [12:09:50,690][INFO][main][] Direct IO is enabled for block IO operations on aligned memory structures. [block size = 4096, durable memory page size = 4096] [12:09:50,694][INFO][main][IgniteKernal] Non-loopback local IPs: 192.168.1.8 [12:09:50,694][INFO][main][IgniteKernal] Enabled local MACs: 6CAE8B663824 [12:09:50,701][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276785084-40a29af6-4442-4caf-bdea-0e839274c6cb-START.bin, endMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276670048-e51bcb86-1f33-4bb9-bb3b-332860432fb9-END.bin] [12:09:50,712][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24792, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [12:09:50,714][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=62256, len=1963], lastMarked=FileWALPointer [idx=0, fileOff=166147, len=21409], lastCheckpointId=40a29af6-4442-4caf-bdea-0e839274c6cb] [12:09:50,843][INFO][main][GridCacheDatabaseSharedManager] Found last checkpoint marker [cpId=40a29af6-4442-4caf-bdea-0e839274c6cb, pos=FileWALPointer [idx=0, fileOff=166147, len=21409]] [12:09:50,912][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=166147, len=21409], lastCheckpointId=40a29af6-4442-4caf-bdea-0e839274c6cb] [12:09:50,958][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=50 ms] [12:09:50,958][INFO][main][GridCacheProcessor] Restoring partition state for local groups. [12:09:50,959][INFO][main][GridCacheProcessor] Finished restoring partition state for local groups [groupsProcessed=0, partitionsProcessed=0, time=0ms] [12:09:51,004][INFO][main][GridClusterStateProcessor] Restoring history for BaselineTopology[id=0] [12:09:51,057][INFO][main][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'null' to 'false' [12:09:51,058][INFO][main][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustTimeout' was changed from 'null' to '300000' [12:09:51,059][INFO][main][IgniteH2Indexing] SQL parameter 'sql.disabledFunctions' was changed from 'null' to '[FILE_WRITE, CANCEL_SESSION, MEMORY_USED, CSVREAD, LINK_SCHEMA, MEMORY_FREE, FILE_READ, CSVWRITE, SESSION_ID, LOCK_MODE]' [12:09:51,084][INFO][main][FilePageStoreManager] Cleanup cache stores [total=1, left=0, cleanFiles=false] [12:09:51,103][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24792, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [12:09:51,104][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24792, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [12:09:51,106][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24792, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [12:09:51,108][INFO][main][GridCacheDatabaseSharedManager] Configured data regions started successfully [total=4] [12:09:51,108][INFO][main][GridCacheDatabaseSharedManager] Starting binary memory restore for: [-2100569601, 1499592267] [12:09:51,195][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276785084-40a29af6-4442-4caf-bdea-0e839274c6cb-START.bin, endMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276670048-e51bcb86-1f33-4bb9-bb3b-332860432fb9-END.bin] [12:09:51,196][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=62256, len=1963], lastMarked=FileWALPointer [idx=0, fileOff=166147, len=21409], lastCheckpointId=40a29af6-4442-4caf-bdea-0e839274c6cb] [12:09:51,238][WARNING][main][GridCacheDatabaseSharedManager] Ignite node stopped in the middle of checkpoint. Will restore memory state and finish checkpoint on node start. [12:09:51,297][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=6.3 GiB, pages=1595760, tableSize=124.1 MiB, checkpointBuffer=1.6 GiB] [12:09:51,400][INFO][main][GridCacheDatabaseSharedManager] Found last checkpoint marker [cpId=40a29af6-4442-4caf-bdea-0e839274c6cb, pos=FileWALPointer [idx=0, fileOff=166147, len=21409]] [12:09:51,408][INFO][main][GridCacheDatabaseSharedManager] Finished applying memory changes [changesApplied=25, time=111 ms] [12:09:51,683][INFO][main][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=40a29af6-4442-4caf-bdea-0e839274c6cb, pages=16, markPos=FileWALPointer [idx=0, fileOff=166147, len=21409], pagesWrite=200ms, fsync=20ms, total=220ms] [12:09:51,686][INFO][main][GridCacheDatabaseSharedManager] Binary memory state restored at node startup [restoredPtr=FileWALPointer [idx=0, fileOff=187556, len=0]] [12:09:51,687][INFO][main][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/usr/share/apache-ignite/work/db/wal/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/0000000000000000.wal, offset=187556, ver=2] [12:09:51,813][INFO][main][GridCacheProcessor] Started cache in recovery mode [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [12:09:51,879][INFO][main][GridCacheProcessor] Started cache in recovery mode [name=SQL_PUBLIC_CT_COUNTRIES, id=1499592267, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=0, mvcc=false] [12:09:51,889][INFO][main][GridCacheDatabaseSharedManager] Binary recovery performed in 781 ms. [12:09:51,892][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276785084-40a29af6-4442-4caf-bdea-0e839274c6cb-START.bin, endMarker=/usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/cp/1594276785084-40a29af6-4442-4caf-bdea-0e839274c6cb-END.bin] [12:09:51,893][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=166147, len=21409], lastCheckpointId=40a29af6-4442-4caf-bdea-0e839274c6cb] [12:09:51,931][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=31 ms] [12:09:51,932][INFO][main][GridCacheProcessor] Restoring partition state for local groups. [12:09:52,070][INFO][main][GridCacheProcessor] Finished restoring partition state for local groups [groupsProcessed=2, partitionsProcessed=1124, time=131ms] [12:09:52,132][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [12:09:53,384][INFO][disco-notifier-worker-#83][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], nodeId=29a661dd-7405-420d-a230-3250142f10c1, ver=1594276793371, local=true, initialized=false]] [12:09:53,400][INFO][disco-notifier-worker-#83][GridClusterStateProcessor] Sending activate request with BaselineTopology BaselineTopology [id=0, branchingHash=-1658423201, branchingType='Cluster activation', baselineNodes=[e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d]] [12:09:53,411][INFO][disco-notifier-worker-#83][GridClusterStateProcessor] Received activate request with BaselineTopology[id=0] [12:09:53,414][INFO][disco-notifier-worker-#83][GridClusterStateProcessor] Started state transition: true [12:09:53,464][INFO][exchange-worker-#85][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=29a661dd-7405-420d-a230-3250142f10c1, customEvt=ChangeGlobalStateMessage [id=aca3d423371-c70c9dd8-4097-4051-8fd5-369fe12071ad, reqId=6746ece9-059e-4cf8-81dc-cf92a2fd02ca, initiatingNodeId=29a661dd-7405-420d-a230-3250142f10c1, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-1658423201, branchingType='Cluster activation', baselineNodes=[e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d]], forceChangeBaselineTopology=false, timestamp=1594276793400], allowMerge=false, exchangeFreeSwitch=false] [12:09:53,464][INFO][exchange-worker-#85][GridDhtPartitionsExchangeFuture] Start activation process [nodeId=29a661dd-7405-420d-a230-3250142f10c1, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [12:09:53,465][INFO][exchange-worker-#85][FilePageStoreManager] Resolved page store work directory: /usr/share/apache-ignite/work/db/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d [12:09:53,467][INFO][exchange-worker-#85][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/usr/share/apache-ignite/work/db/wal/node00-e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d/0000000000000000.wal, offset=187585, ver=2] [12:09:53,468][INFO][exchange-worker-#85][GridClusterStateProcessor] Writing BaselineTopology[id=0] [12:09:53,480][INFO][exchange-worker-#85][GridCacheDatabaseSharedManager] Finish recovery performed in 14 ms. [12:09:53,480][INFO][exchange-worker-#85][msg] Components activation performed in 15 ms. [12:09:53,492][SEVERE][exchange-worker-#85][GridDhtPartitionsExchangeFuture] Failed to activate node components [nodeId=29a661dd-7405-420d-a230-3250142f10c1, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] java.lang.IllegalStateException: Duplicate key at org.apache.ignite.cache.QueryEntity.checkIndexes(QueryEntity.java:233) at org.apache.ignite.cache.QueryEntity.makePatch(QueryEntity.java:184) at org.apache.ignite.internal.processors.query.QuerySchema.makePatch(QuerySchema.java:114) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1972) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$null$6a5b31b9$1(GridCacheProcessor.java:1830) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCachesIfPossible$9(GridCacheProcessor.java:1771) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$926b6886$1(GridCacheProcessor.java:1827) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11157) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11059) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1822) at org.apache.ignite.internal.processors.cache.GridCacheProcessor.prepareStartCachesIfPossible(GridCacheProcessor.java:1769) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.processCacheStartRequests(CacheAffinitySharedManager.java:965) at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:851) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:1182) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:815) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) [12:09:53,504][SEVERE][exchange-worker-#85][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitions (rebalancing will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=ChangeGlobalStateMessage [id=aca3d423371-c70c9dd8-4097-4051-8fd5-369fe12071ad, reqId=6746ece9-059e-4cf8-81dc-cf92a2fd02ca, initiatingNodeId=29a661dd-7405-420d-a230-3250142f10c1, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-1658423201, branchingType='Cluster activation', baselineNodes=[e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d]], forceChangeBaselineTopology=false, timestamp=1594276793400], affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=29a661dd-7405-420d-a230-3250142f10c1, consistentId=e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:48550], discPort=48550, order=1, intOrder=1, lastExchangeTime=1594276792124, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false], topVer=1, nodeId8=29a661dd, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594276793411]], nodeId=29a661dd, evt=DISCOVERY_CUSTOM_EVT] java.lang. at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1483) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:894) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) [12:09:53,505][INFO][exchange-worker-#85][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=null, err=java.lang.NullPointerException, rebalanced=false, wasRebalanced=false] [12:09:53,508][INFO][exchange-worker-#85][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=29a661dd-7405-420d-a230-3250142f10c1, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=29a661dd-7405-420d-a230-3250142f10c1, consistentId=e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:48550], discPort=48550, order=1, intOrder=1, lastExchangeTime=1594276792124, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false], done=true, newCrdFut=null], topVer=null] [12:09:53,513][SEVERE][exchange-worker-#85][GridCachePartitionExchangeManager] Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=29a661dd-7405-420d-a230-3250142f10c1, consistentId=e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:48550], discPort=48550, order=1, intOrder=1, lastExchangeTime=1594276792124, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false], topVer=1, nodeId8=29a661dd, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594276793411]], crd=TcpDiscoveryNode [id=29a661dd-7405-420d-a230-3250142f10c1, consistentId=e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:48550], discPort=48550, order=1, intOrder=1, lastExchangeTime=1594276792124, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=29a661dd-7405-420d-a230-3250142f10c1, consistentId=e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:48550], discPort=48550, order=1, intOrder=1, lastExchangeTime=1594276792124, loc=true, ver=2.8.1#20200521-sha1:86422096, isClient=false], topVer=1, nodeId8=29a661dd, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1594276793411]], nodeId=29a661dd, evt=DISCOVERY_CUSTOM_EVT], added=true, exchangeType=ALL, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=2079879925], init=false, lastVer=null, partReleaseFut=null, exchActions=ExchangeActions [startCaches=[ignite-sys-cache, SQL_PUBLIC_CT_COUNTRIES], stopCaches=null, startGrps=[ignite-sys-cache, SQL_PUBLIC_CT_COUNTRIES], stopGrps=[], resetParts=null, stateChangeRequest=StateChangeRequest [msg=ChangeGlobalStateMessage [id=aca3d423371-c70c9dd8-4097-4051-8fd5-369fe12071ad, reqId=6746ece9-059e-4cf8-81dc-cf92a2fd02ca, initiatingNodeId=29a661dd-7405-420d-a230-3250142f10c1, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-1658423201, branchingType='Cluster activation', baselineNodes=[e04f1d98-0586-4b4d-ae4b-eef7d6a4c07d]], forceChangeBaselineTopology=false, timestamp=1594276793400], prevBltHistItem=o.a.i.i.processors.cluster.BaselineTopologyHistoryItem@64911ad3, activeChanged=true, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]], affChangeMsg=null, centralizedAff=false, forceAffReassignment=true, exchangeLocE=java.lang.IllegalStateException: Duplicate key, cacheChangeFailureMsgSent=false, done=true, state=CRD, registerCachesFuture=null, partitionsSent=false, partitionsReceived=false, delayedLatestMsg=null, afterLsnrCompleteFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=444367653], timeBag=o.a.i.i.util.TimeBag@4a3d6054, startTime=142732262291967, initTime=1594276793462, rebalanced=false, evtLatch=0, remaining=HashSet [], mergedJoinExchMsgs=null, awaitMergedMsgs=0, super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=java.lang.NullPointerException, hash=46874807]] class org.apache.ignite.IgniteCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7507) at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3242) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1483) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:894) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214) ... 3 more [12:09:53,515][SEVERE][exchange-worker-#85][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: null]] class org.apache.ignite.IgniteCheckedException: null at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7507) at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:260) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:209) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3242) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3063) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.NullPointerException at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1483) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:894) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3214) ... 3 more [12:09:53,519][INFO][main][IgniteKernal] Performance suggestions for grid (fix if possible) [12:09:53,519][INFO][main][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [12:09:53,520][INFO][main][IgniteKernal] ^-- Disable fully synchronous writes (set 'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC) [12:09:53,519][WARNING][exchange-worker-#85][CacheDiagnosticManager] Page locks dump: Thread=[name=dms-writer-thread-#89, id=126], state=WAITING Locked pages = [] Locked pages log: name=dms-writer-thread-#89 time=(1594276793516, 2020-07-09 12:09:53.516) Thread=[name=exchange-worker-#85, id=122], state=RUNNABLE Locked pages = [] Locked pages log: name=exchange-worker-#85 time=(1594276793516, 2020-07-09 12:09:53.516) Thread=[name=main, id=1], state=RUNNABLE Locked pages = [] Locked pages log: name=main time=(1594276793516, 2020-07-09 12:09:53.516) [12:09:53,520][INFO][main][IgniteKernal] ^-- Switch to the most recent 1.8 JVM version [12:09:53,520][INFO][main][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=[g|G|m|M|k|K]' to JVM options) [12:09:53,520][INFO][main][IgniteKernal] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [12:09:53,520][SEVERE][exchange-worker-#85][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteCheckedException: null]] [12:09:53,520][INFO][main][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [12:09:53,520][INFO][main][IgniteKernal] [12:09:53,520][INFO][main][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}