Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10128

IO race during read\write cache configurations.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6
    • Fix Version/s: 2.8
    • Component/s: persistence
    • Labels:
      None

      Description

      During grid usage, found problem below. I tries simultaneously start several nodes and call activate on ** each node.

      [15:38:25,186][INFO][exchange-worker-#63][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/wal/node2/0000000000000000.wal, offset=0, ver=2]
      [15:38:25,210][INFO][exchange-worker-#63][GridSnapshotAwareClusterStateProcessorImpl] Writing BaselineTopology[id=0]
      [15:38:25,225][INFO][exchange-worker-#63][GridCacheDatabaseSharedManager] Resume logging performed in 41 ms.
      [15:38:25,252][INFO][exchange-worker-#63][CacheAffinitySharedManager] Updating caches registry performed in 27 ms.
      [15:38:25,743][SEVERE][sys-#70][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp]]
      Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp]]
      java.nio.file.NoSuchFileException: /storage/ssd/mshonichev/tiden/pme-181030-153724/test_pme_bench/ignite.server.2/work/db/node2/cacheGroup-cache_group_2/cache_group_1_010cache_data.dat.tmp
          at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
          at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
          at java.nio.file.Files.move(Files.java:1395)
          at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.storeCacheData(FilePageStoreManager.java:390)
          at org.apache.ignite.internal.processors.cache.CachesRegistry.lambda$persistCacheConfigurations$4(CachesRegistry.java:303)
          at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6875)
          at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
          at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zstan Stanilovsky Evgeny
                Reporter:
                zstan Stanilovsky Evgeny
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: