Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-8435

Inconsistent data after StopNodeFailureHandler handles process termination

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5
    • Component/s: None
    • Labels:
      None

      Description

      Steps to reproduce:

      1. Cluster started with StopNodeFailureHandler in config.
      2. Upload data to the caches and run accounts task (transfer the amount from one account to another for some cache).
      3. Terminate one thread (ttl-cleanup-worker for example for one node).
      4. Wait until accounts task finished the work.
      5. Check the sum for caches.
      Expected result:
      The sum is the same as before thread rermination.
      Actual result:
      The sum is not the same for some caches.

      Also, I see these exception in the logs for node with terminated thread:

      [13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
      class org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Commit produced a runtime exception (all transaction entries will be invalidated): GridDhtTxRemote[xid=8e4906dd261-00000000-0815-47fa-0000-000000000004, xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
      eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ, state=COMMITTING, invalidate=false, rollbackOnly=false, nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
       at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
       at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
       at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
       at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
       at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
       at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
       at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on search row: org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
       at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3604)
       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3580)
       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1040)
       at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:611)
       ... 18 more
      

       

        Attachments

          Activity

            People

            • Assignee:
              akalashnikov Anton Kalashnikov
              Reporter:
              akalashnikov Anton Kalashnikov
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m