[IGNITE-10128] IO race during read\write cache configurations.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6
• Fix Version/s: 2.8
• Component/s: persistence
• Assignee: Evgeny Stanilovsky
• Reporter: Evgeny Stanilovsky
• Created: 02/Nov/18 11:09
• Updated: 09/Nov/18 09:03
• Resolved: 09/Nov/18 09:00

Description

During grid usage, found problem below. I tries simultaneously start several nodes and call activate on ** each node.

[15:38:25,186][INFO][exchange-worker-#63][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/wal/node2/0000000000000000.wal, offset=0, ver=2]
[15:38:25,210][INFO][exchange-worker-#63][GridSnapshotAwareClusterStateProcessorImpl] Writing BaselineTopology[id=0]
[15:38:25,225][INFO][exchange-worker-#63][GridCacheDatabaseSharedManager] Resume logging performed in 41 ms.
[15:38:25,252][INFO][exchange-worker-#63][CacheAffinitySharedManager] Updating caches registry performed in 27 ms.
[15:38:25,743][SEVERE][sys-#70][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp]]
Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp]]
java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.storeCacheData(FilePageStoreManager.java:390)
    at org.apache.ignite.internal.processors.cache.CachesRegistry.lambda$persistCacheConfigurations$4(CachesRegistry.java:303)
    at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6875)
    at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #5241