Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-5757

RENT partition state is set before exchange is completed

    XMLWordPrintableJSON

Details

    Description

      I observe the following error on TC during GridCachePartitionedGetAndTransformStoreSelfTest run:

      [2017-07-14 16:26:49,857][ERROR][sys-stripe-3-#4%near.GridCachePartitionedGetAndTransformStoreSelfTest%][IgniteTxHandler] Failed to prepare DHT transaction: GridDhtTxLocal [nearNodeId=8a4cdb3f-85be-4286-a63d-728a6c800001, nearFutId=a8ef7414d51-e91dff67-b46c-4762-b7b9-9bcf382a82a0, nearMiniId=1, nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion [topVer=111518810, order=1500038810278, nodeOrder=3], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=true, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=239, startTime=1500038809744, nodeId=322e96c3-eb1c-4aea-b9fd-3a3dae368d31, startVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], finalizing=NONE, invalidParts=null, state=ROLLED_BACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], duration=107ms, onePhaseCommit=false], size=0]]]
      class org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: Failed to prepare transaction: GridDhtTxLocal [nearNodeId=8a4cdb3f-85be-4286-a63d-728a6c800001, nearFutId=a8ef7414d51-e91dff67-b46c-4762-b7b9-9bcf382a82a0, nearMiniId=1, nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion [topVer=111518810, order=1500038810278, nodeOrder=3], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=true, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=239, startTime=1500038809744, nodeId=322e96c3-eb1c-4aea-b9fd-3a3dae368d31, startVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=MARKED_ROLLBACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], duration=97ms, onePhaseCommit=false], size=0]]]
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:418)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:459)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:121)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:95)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:141)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:139)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
      	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
      	at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: class org.apache.ignite.IgniteCheckedException: Adding entry to partition that is concurrently evicted [part=58, shouldBeMoving=false]
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.addEntry(GridDhtTxLocalAdapter.java:523)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:399)
      	... 17 more
      

      The issue happens because GridDhtLocalTransaction is being prepared on topology version (4, 0), and the next topology version (4, 1) changes the primary node assignment. The topology is locked on the originating node, which is a server node, at the same time topology is not locked on primary node that is about to rent it's partitions.
      When exchange runs on primary node, it does not see the GridDhtTxLocal transaction yet, completes the partition release future and rents all partitions.
      After this, the GridDhtTxLocat starts prepare phase and gets the exception above.

      Attachments

        Activity

          People

            agoncharuk Alexey Goncharuk
            agoncharuk Alexey Goncharuk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: