[2022-08-18T12:37:41,319][INFO ][RMI TCP Connection(103)-10.255.12.2][IgniteSnapshotManager] Cluster-wide snapshot operation started [snpName=snap_2, grps=[Deimos, Phobos, MetaStorage]]
[2022-08-18T12:37:41,320][INFO ][disco-notifier-worker-#89][IgniteSnapshotManager] Snapshot task has been registered on local node [sctx=GridCacheSharedManagerAdapter [starting=true, stop=false], task=SnapshotFutureTask, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3]]
[2022-08-18T12:37:41,320][INFO ][exchange-worker-#93][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=c6e8f636-8780-4eb6-85f1-20d3fad4eaf7, customEvt=SnapshotStartDiscoveryMessage [super=InitMessage [id=a737840b281-a0d1da48-0004-44ba-8696-00bd356b433f, processId=6ce06231-7239-4f08-9d3c-9b5b2acf9829, type=3]], allowMerge=false, exchangeFreeSwitch=true]
[2022-08-18T12:37:41,321][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
[2022-08-18T12:37:41,323][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3]]]]
[2022-08-18T12:37:41,323][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], waitTime=0ms, futInfo=NA, mode=LOCAL]
[2022-08-18T12:37:41,328][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], err=null, rebalanced=true, wasRebalanced=true]
[2022-08-18T12:37:41,328][INFO ][exchange-worker-#93][AbstractSnapshotFutureTask] Resolved directory for snapshotting cache group: /test/gav/Ignite-SE-4.2131.0/work/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/snp/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/cache-Deimos
[2022-08-18T12:37:41,328][INFO ][exchange-worker-#93][AbstractSnapshotFutureTask] Resolved directory for snapshotting cache group: /test/gav/Ignite-SE-4.2131.0/work/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/snp/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/cache-Phobos
[2022-08-18T12:37:41,328][INFO ][exchange-worker-#93][AbstractSnapshotFutureTask] Resolved directory for snapshotting metastorage: /test/gav/Ignite-SE-4.2131.0/work/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/snp/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/metastorage
[2022-08-18T12:37:41,328][INFO ][exchange-worker-#93][AbstractSnapshotFutureTask] Snapshot operation is scheduled on local node and will be handled by the checkpoint listener [sctx=SnapshotFutureTask [pageStore=GridCacheSharedManagerAdapter [starting=true, stop=false], locBuff=java.lang.ThreadLocal$SuppliedThreadLocal@52d6375c, withMetaStorage=true, cpEndFut=java.util.concurrent.CompletableFuture@49f4d3c8[Not completed], startedFut=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1513170808], tmpConsIdDir=/test/gav/Ignite-SE-4.2131.0/work/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d/snp/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d, closeFut=null, started=true, totalSize=0, processedSize=0, super=AbstractSnapshotFutureTask []], topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3]]
[2022-08-18T12:37:41,330][INFO ][db-checkpoint-thread-#103][AbstractSnapshotFutureTask] Finished waiting for all the concurrent operations over the metadata store before snapshot [snpName=snap_2, time=0ms]
[2022-08-18T12:37:41,342][ERROR][db-checkpoint-thread-#103][AbstractSnapshotFutureTask] Snapshot task has accepted exception to stop
org.apache.ignite.IgniteException: Failed to create snapshot work directory for a local snapshot sender: /bad/path/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$LocalSnapshotSender.init(IgniteSnapshotManager.java:3133) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$onCheckpointBegin$1(SnapshotFutureTask.java:443) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$wrapExceptionIfStarted$8(SnapshotFutureTask.java:584) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.onCheckpointBegin(SnapshotFutureTask.java:444) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointWorkflow.markCheckpointBegin(CheckpointWorkflow.java:320) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.doCheckpoint(Checkpointer.java:419) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.body(Checkpointer.java:277) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.apache.ignite.IgniteCheckedException: Failed to create snapshot work directory for a local snapshot sender: /bad/path/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d
	at org.apache.ignite.internal.util.IgniteUtils.ensureDirectory(IgniteUtils.java:9971) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$LocalSnapshotSender.init(IgniteSnapshotManager.java:3130) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	... 8 more
[2022-08-18T12:37:41,342][ERROR][exchange-worker-#93][IgniteSnapshotManager] Fail to wait while cluster-wide snapshot operation started
org.apache.ignite.IgniteCheckedException: Failed to create snapshot work directory for a local snapshot sender: /bad/path/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d
	at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7865) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:261) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:192) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:142) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.onDoneBeforeTopologyUnlock(IgniteSnapshotManager.java:1639) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2534) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:158) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:532) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:1096) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3369) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3200) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.apache.ignite.IgniteException: Failed to create snapshot work directory for a local snapshot sender: /bad/path/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$LocalSnapshotSender.init(IgniteSnapshotManager.java:3133) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$onCheckpointBegin$1(SnapshotFutureTask.java:443) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$wrapExceptionIfStarted$8(SnapshotFutureTask.java:584) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.onCheckpointBegin(SnapshotFutureTask.java:444) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointWorkflow.markCheckpointBegin(CheckpointWorkflow.java:320) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.doCheckpoint(Checkpointer.java:419) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.body(Checkpointer.java:277) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	... 2 more
Caused by: org.apache.ignite.IgniteCheckedException: Failed to create snapshot work directory for a local snapshot sender: /bad/path/snap_2/db/node00-ec1155ad-0533-4e54-859b-26125a34a63d
	at org.apache.ignite.internal.util.IgniteUtils.ensureDirectory(IgniteUtils.java:9971) [ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$LocalSnapshotSender.init(IgniteSnapshotManager.java:3130) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$onCheckpointBegin$1(SnapshotFutureTask.java:443) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.lambda$wrapExceptionIfStarted$8(SnapshotFutureTask.java:584) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTask.onCheckpointBegin(SnapshotFutureTask.java:444) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.CheckpointWorkflow.markCheckpointBegin(CheckpointWorkflow.java:320) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.doCheckpoint(Checkpointer.java:419) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	at org.apache.ignite.internal.processors.cache.persistence.checkpoint.Checkpointer.body(Checkpointer.java:277) ~[ignite-core-2.13.1-p0.jar:2.13.1-p0]
	... 2 more
[2022-08-18T12:37:41,348][INFO ][db-checkpoint-thread-#103][Checkpointer] Checkpoint started [checkpointId=203c698d-550f-4db2-b9d4-5f6ee81e9419, startPtr=WALPointer [idx=0, fileOff=374548, len=40871], checkpointBeforeLockTime=2ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=7ms, checkpointLockHoldTime=10ms, walCpRecordFsyncDuration=2ms, writeCheckpointEntryDuration=2ms, splitAndSortCpPagesDuration=0ms, pages=3, reason='Start snapshot operation: snap_2']
[2022-08-18T12:37:41,350][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=c6e8f636-8780-4eb6-85f1-20d3fad4eaf7, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=c6e8f636-8780-4eb6-85f1-20d3fad4eaf7, consistentId=ec1155ad-0533-4e54-859b-26125a34a63d, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 10.40.6.204, 127.0.0.1], sockAddrs=HashSet [devmon-4/10.40.6.204:49500, /0:0:0:0:0:0:0:1%lo:49500, /127.0.0.1:49500], discPort=49500, order=1, intOrder=1, lastExchangeTime=1660815461347, loc=true, ver=2.13.1#20220629-sha1:7af95931, isClient=false], rebalanced=true, done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=2, minorTopVer=3]]
[2022-08-18T12:37:41,350][INFO ][exchange-worker-#93][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], resVer=AffinityTopologyVersion [topVer=2, minorTopVer=3], stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters initialization" (0 ms), stage="Determine exchange type" (0 ms), stage="Preloading notification" (0 ms), stage="Wait partitions release latch [latch=exchange]" (2 ms), stage="After states restored callback" (0 ms), stage="WAL history reservation" (0 ms), stage="Exchange done" (26 ms), stage="Total time" (28 ms)]