Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-5757

RENT partition state is set before exchange is completed

    XMLWordPrintableJSON

    Details

      Description

      I observe the following error on TC during GridCachePartitionedGetAndTransformStoreSelfTest run:

      [2017-07-14 16:26:49,857][ERROR][sys-stripe-3-#4%near.GridCachePartitionedGetAndTransformStoreSelfTest%][IgniteTxHandler] Failed to prepare DHT transaction: GridDhtTxLocal [nearNodeId=8a4cdb3f-85be-4286-a63d-728a6c800001, nearFutId=a8ef7414d51-e91dff67-b46c-4762-b7b9-9bcf382a82a0, nearMiniId=1, nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion [topVer=111518810, order=1500038810278, nodeOrder=3], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=true, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=239, startTime=1500038809744, nodeId=322e96c3-eb1c-4aea-b9fd-3a3dae368d31, startVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], finalizing=NONE, invalidParts=null, state=ROLLED_BACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], duration=107ms, onePhaseCommit=false], size=0]]]
      class org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: Failed to prepare transaction: GridDhtTxLocal [nearNodeId=8a4cdb3f-85be-4286-a63d-728a6c800001, nearFutId=a8ef7414d51-e91dff67-b46c-4762-b7b9-9bcf382a82a0, nearMiniId=1, nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion [topVer=111518810, order=1500038810278, nodeOrder=3], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=true, nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], writeVer=null, implicit=true, loc=true, threadId=239, startTime=1500038809744, nodeId=322e96c3-eb1c-4aea-b9fd-3a3dae368d31, startVer=GridCacheVersion [topVer=111518810, order=1500038810305, nodeOrder=1], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, invalidParts=null, state=MARKED_ROLLBACK, timedOut=false, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], duration=97ms, onePhaseCommit=false], size=0]]]
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:418)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareNearTx(IgniteTxHandler.java:459)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxPrepareRequest(IgniteTxHandler.java:121)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$000(IgniteTxHandler.java:95)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:141)
      	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$1.apply(IgniteTxHandler.java:139)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
      	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
      	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
      	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
      	at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: class org.apache.ignite.IgniteCheckedException: Adding entry to partition that is concurrently evicted [part=58, shouldBeMoving=false]
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.addEntry(GridDhtTxLocalAdapter.java:523)
      	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.prepareAsync(GridDhtTxLocal.java:399)
      	... 17 more
      

      The issue happens because GridDhtLocalTransaction is being prepared on topology version (4, 0), and the next topology version (4, 1) changes the primary node assignment. The topology is locked on the originating node, which is a server node, at the same time topology is not locked on primary node that is about to rent it's partitions.
      When exchange runs on primary node, it does not see the GridDhtTxLocal transaction yet, completes the partition release future and rents all partitions.
      After this, the GridDhtTxLocat starts prepare phase and gets the exception above.

        Attachments

          Activity

            People

            • Assignee:
              agoncharuk Alexey Goncharuk
              Reporter:
              agoncharuk Alexey Goncharuk
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: