[CASSANDRA-17813] Scrub/Compaction throws "AssertionError: Illegal bounds …", reads throws "IllegalArgumentException: Unable to seek to position … in read-only mode"

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.11.x
• Component/s: Local/SSTable
• Bug Category: Availability - Response Crash
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned
• Reporter: Michael Semb Wever
• Created: 11/Aug/22 21:14
• Updated: 11/Aug/22 21:18

Description

Compactions and scrub fail with

java.lang.AssertionError: Illegal bounds [1698255698..792484184); size: 1039216223
        at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
        at org.apache.cassandra.io.util.Memory.setByteBuffer(Memory.java:420)
        at org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:164)
        at org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
        at org.apache.cassandra.io.sstable.format.SSTableReader.getIndexScanPosition(SSTableReader.java:1268)
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.seekToCurrentRangeStart(BigTableScanner.java:202)
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.access$400(BigTableScanner.java:52)
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:329)
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:306)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:285)
        at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:463)
        at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:163)
        at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92)
        at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:238)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:199)

and reads fail with

java.lang.IllegalArgumentException: Unable to seek to position 3366498330815836469 in …/md-120354-big-Index.db (133472342807 bytes) in read-only mode
 at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:205) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:207) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1586) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:64) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.initializeIterator(UnfilteredRowIteratorWithLowerBound.java:108) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.partitionLevelDeletion(LazilyInitializedUnfilteredRowIterator.java:81) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.rows.UnfilteredRowIteratorWithLowerBound.partitionLevelDeletion(UnfilteredRowIteratorWithLowerBound.java:170) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:767) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache(SinglePartitionReadCommand.java:581) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:503) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:423) ~[apache-cassandra-3.11.6.jar:3.11.6]
 at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.11.6.jar:3.11.6]

Workaround: stop the node, delete the -Summary.db file, start the node.
e.g. in the above read exception, delete the md-120354-big-Summary.db file while the node is offline.