[IGNITE-20821] Sporadic exception on KeyValueView#get

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0
• Component/s: cache, clients
• Labels: ignite-3 ignite3_performance
• Assignee: Ivan Bessonov
• Reporter: Ivan Artiukhov
• Reviewer: Roman Puchkovskiy
• Created: 09/Nov/23 12:33
• Updated: 04/Sep/24 16:08
• Resolved: 13/Nov/23 07:39

Description

AI3, rev. 8ebd9e842a2aaec5e1ae7e3b5e1cb36a040dc565

Benchmark: https://github.com/gridgain/YCSB/blob/ycsb-2023.7/ignite3/src/main/java/site/ycsb/db/ignite3/IgniteClient.java 

Cluster setup: 1 server node, aipersist, 25 partitions, raft.fsync=false

Steps

Run the mentioned key-value benchmark to preload data (10 million unique keys):

 

Command line: -db site.ycsb.db.ignite3.IgniteClient -load -P /opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p recordcount=250000 -p warmupops=50000 -p dataintegrity=true -p measurementtype=timeseries -p status.interval=1 -p hosts=192.168.1.63 -p recordcount=10000000 -p operationcount=10000000 -s

Observe that all 10M keys have been loaded:

 

 

[INSERT], Return=OK, 10000000

 Run the 100% get operations over preloaded data:

 

Command line: -db site.ycsb.db.ignite3.IgniteClient -t -P /opt/pubagent/poc/config/ycsb/workloads/workloadc -threads 1 -p operationcount=250000 -p recordcount=250000 -p warmupops=50000 -p dataintegrity=true -p measurementtype=timeseries -p status.interval=1 -p hosts=192.168.1.63 -p recordcount=10000000 -p operationcount=10000000 -s

Expected result

All 10M keys were read without errors

Actual result

Reading 4341 keys out of 10M keys failed with exception like the following one:

Caused by: org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 Row version lookup failed: [rowId=RowId [partitionId=2, uuid=c18be8d8-e29d-4be9-9238-e85a8a18a3b9], tableId=5, partitionId=2]
	at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:879)
	at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.lambda$read$8(AbstractPageMemoryMvPartitionStorage.java:332)
	at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:749)
	at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.read(AbstractPageMemoryMvPartitionStorage.java:324)
	at org.apache.ignite.internal.table.distributed.replicator.PartitionReplicaListener.resolveRowByPkForReadOnly(PartitionReplicaListener.java:1843)
	... 43 more
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:069639f9-b2b9-4690-b066-3ca0bf2e9e06 B+Tree is corrupted [groupId=5, pageIds=[562958543400296], groupName=5, msg=Runtime failure on lookup row: org.apache.ignite.internal.storage.pagememory.mv.VersionChainKey@1b8df27b]
	at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6724)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1580)
	at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.findVersionChain(AbstractPageMemoryMvPartitionStorage.java:870)
	... 47 more
Caused by: java.lang.AssertionError
	at org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6464)
	at org.apache.ignite.internal.pagememory.tree.BplusTree$Search.run(BplusTree.java:369)
	at org.apache.ignite.internal.pagememory.tree.BplusTree$GetPageHandler.run(BplusTree.java:6451)
	at org.apache.ignite.internal.pagememory.util.PageHandler.readPage(PageHandler.java:157)
	at org.apache.ignite.internal.pagememory.datastructure.DataStructure.read(DataStructure.java:417)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.read(BplusTree.java:6693)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1654)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.findDown(BplusTree.java:1663)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.doFind(BplusTree.java:1632)
	at org.apache.ignite.internal.pagememory.tree.BplusTree.findOne(BplusTree.java:1572)
	... 48 more

Issue Links

• links to: GitHub Pull Request #2829