[IGNITE-11291] Assertion error in time of rebalance completion lead to to critical failure node

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Alexey Scherbakov
• Reporter: Vladislav Pyatkov
• Created: 11/Feb/19 14:29
• Updated: 25/Mar/19 10:12

Description

java.lang.AssertionError: Got removed exception on entry with dht local candidate: [IgniteTxEntry [key=KeyCacheObjectImpl [part=11859, val=3338011748811769508, hasValBytes=true], cacheId=-313938805, txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=11859, val=3338011748811769508, hasValBytes=true], cacheId=-313938805], val=[op=UPDATE, val=com.sbt.bm.ucp.common.dpl.model.party.hashcodes.DPartyHashCode_DPL_PROXY [idHash=1985166276, hash=1530579783, colocationKey=11859, sync Flag=null, hashUpdateTime=Sat Feb 09 20:16:55 MSK 2019, lastChangeDate=1549732615042, partition_DPL_id=4, ownerId=ucp, externalSystem=21, serializedHashCodesMap={"Address":["581a1367d4f50172c41168747c99105df1d3a4c6","24d3413bc5d3151dd784e088eaf4603e8e86c40b","fd7ce1449cd539a5976d358fb15060820c630f36"],"BirthDate":["a5cc7c1ac9821a6b19a455ae7eac55f4a4475bd7","415fe3810f8c4555a7188fe62275b34cdd1384cd","1311875998ef2aaccc5860ac6f991107ad4fa558"],"BirthPlace":["2fb5ea4cf7e9cfbdf77baae18f26804a10f51d57","76ddc5f2d959d27044c180957d9d0aed7369c0a2"],"Gender":["ad99678bb0e6de91d6a2ce620ba4fa98206aa9b9"],"IndividualIdentification":["c5b73f0087461fa4b980362344d38afddd1677b6","157952428fbe8c5b80da4db12c945cb4ad1f33f8","4d2257f62b1aa9cf290728d147776dd569b226a7"],"IndividualName":["256dba12a3b9d95dc1ada524d1a67cb590eb3ec2","2302eda39fb8f3751f3646542ad54ae717f5bc2c","9b8084de661530d5dc6ea140df793f4aa417a114"],"PartyToPartyGroup":["8d41984edf2e9ff916da0a74b884554cc2fceca9"],"PhoneNumber":["c89cbfb741ba5ea0fa13091a1cc7591c69374c0c","149081529b36fe29bc72bf88c66e51bdab3ae2d7","3c6bc9cc9e47ecd2f79b1506a22799ba69b95227","389830cd794fd6748f46ab7d8d878a2fec75cf63","9d5c3ecd5c951a16745cefd771b79c17bbc8c665","dfd8959391fd18e89505bfde31e0aa9a80513fec"],"Individual":["82dc325513d53990709bafbf1a334b602025ba17","544aab947d3a33787c7feffc93c4a4572e957dca","2728a40ce9759fa7110bc5d0d3c8b5c193210a2d"]}, uid=null, isDeleted=false, isImmutable=false, checksum=null, id=3338011748811769508, externalClientId=75662144, colocationId=1216693183554769678]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=CacheEntryPredicate[] [], filtersPassed=false, filtersSet=false, entry=GridCacheMapEntry [key=KeyCacheObjectImpl [part=11859, val=3338011748811769508, hasValBytes=true], val=null, startVer=1551196730698, ver=GridCacheVersion [topVer=156972865, order=1546089814280, nodeOrder=65], hash=777160356, extras=GridCacheObsoleteEntryExtras [obsoleteVer=GridCacheVersion [topVer=2147483647, order=0, nodeOrder=0]], flags=2]GridDistributedCacheEntry [super=]GridDhtCacheEntry [rdrs=ReaderId[] [], part=11859, super=], prepared=1, locked=false, nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2, partUpdateCntr=0, serReadVer=GridCacheVersion[topVer=156972865, order=1546089814280, nodeOrder=65], xidVer=null]]
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.checkReadConflict(GridDhtTxPrepareFuture.java:1164)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1223)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.access$000(GridDhtTxPrepareFuture.java:109)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture$2.apply(GridDhtTxPrepareFuture.java:701)
        at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture$2.apply(GridDhtTxPrepareFuture.java:696)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onDone(GridDhtForceKeysFuture.java:153)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onDone(GridDhtForceKeysFuture.java:69)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:276)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.init(GridDhtForceKeysFuture.java:217)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$2.apply(GridDhtPreloader.java:510)
        at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader$2.apply(GridDhtPreloader.java:508)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
        at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
        at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache$AffinityReadyFuture.onDone(GridAffinityAssignmentCache.java:850)
        at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache$AffinityReadyFuture.onDone(GridAffinityAssignmentCache.java:834)
        at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
        at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.clientEventTopologyChange(GridAffinityAssignmentCache.java:453)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:1194)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:1190)
        at org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.lambda$forAllCacheGroups$287a3995$1(CacheAffinitySharedManager.java:1277)
        at org.apache.ignite.internal.util.IgniteUtils.lambda$null$1(IgniteUtils.java:10595)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

Issue Links

• duplicates: IGNITE-11618 Assertion got removed exception on entry with dht local candidate on transaction timeout
• is part of: IGNITE-11171 Assertion on tx preparing