[IGNITE-12593] Corruption of B+Tree caused by byte array values and TTL

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8
• Ignite Flags: Docs Required, Release Notes Required
• Assignee: Anton Kalashnikov
• Reporter: Anton Kalashnikov
• Created: 28/Jan/20 12:59
• Updated: 02/Apr/20 09:24
• Resolved: 05/Feb/20 19:06

Description

It seems that the following set of parameters may lead to a corruption of B+Tree:

• persistence is enabled
• TTL is enabled
• Expiry policy - AccessedExpiryPolicy 1 sec.
• cache value type is byte[]
• all caches belong to the same cache group

Example of the stack trace:

[2019-07-16 21:13:19,288][ERROR][sys-stripe-2-#46%db.IgnitePdsWithTtlDeactivateOnHighloadTest1%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1237460590, val2=281586645860358]], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], hash=378, cacheId=-1806498247]]]]
class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: B+Tree is corrupted [pages(groupId, pageId)=[IgniteBiTuple [val1=-1237460590, val2=281586645860358]], msg=Runtime failure on search row: SearchRow [key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], hash=378, cacheId=-1806498247]]
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.corruptedTreeException(BPlusTree.java:5910)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1859)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke0(IgniteCacheOffheapManagerImpl.java:1662)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1645)
	at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:2410)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:445)
	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2309)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2570)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:2030)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1848)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1668)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3235)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:139)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:273)
	at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:268)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1141)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1558)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1186)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
	at org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1083)
	at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:559)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: Invalid entry [entry=AtomicCacheUpdateClosure [entry=GridDhtCacheEntry [rdrs=ReaderId[] [], part=26, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], val=CacheObjectByteArrayImpl [arrLen=1024], ver=GridCacheVersion [topVer=174780794, order=1563300795497, nodeOrder=2], hash=378, extras=GridCacheTtlEntryExtras [ttl=0, expireTime=1563300799721], flags=3]]], topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], newVer=GridCacheVersion [topVer=174780794, order=1563300801949, nodeOrder=2], op=UPDATE, readThrough=false, writeThrough=false, keepBinary=false, expiryPlc=CacheExpiryPolicy [], primary=true, verCheck=false, explicitTtl=-1, explicitExpireTime=-1, conflictVer=null, conflictResolve=true, intercept=false, updateCntr=null, skipInterceptorOnConflict=false, updateRes=null, treeOp=PUT, newRow=null, oldRow=CacheDataRowAdapter [key=KeyCacheObjectImpl [part=26, val=378, hasValBytes=true], val=CacheObjectByteArrayImpl [arrLen=1024], expireTime=1563300799721, ver=GridCacheVersion [topVer=174780794, order=1563300795497, nodeOrder=2], cacheId=-1806498247, link=0001001a0000002a], oldRowExpiredFlag=false, wasIntercepted=false], locNodeId=c0cf7462-d27b-4a46-b157-fb6c02d00001]
	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.update(GridCacheMapEntry.java:6378)
	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:6137)
	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:5828)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3800)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$5700(BPlusTree.java:3694)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1948)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1826)
	... 26 more

The reproducer is attached.

Issue Links

• is blocked by: IGNITE-12594 Deadlock between GridCacheDataStore#purgeExpiredInternal and GridNearTxLocal#enlistWriteEntry
• is related to: IGNITE-12489 Error during purges by expiration: Unknown page type
• is related to: IGNITE-12855 Node failed after get operation when entries from the cache expired concurrently
• links to: GitHub Pull Request #7324
• links to: TC All