[CASSANDRA-15595] Many errors of "java.lang.AssertionError: Illegal bounds"

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21, 3.11.7, 4.0-beta1, 4.0
• Component/s: Local/Caching
• Bug Category: Degradation - Other Exception
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: Linux
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/4886968c4e973488df4f2da480785beff09b562b

Test and Documentation Plan

Added unit test to reproduce.

 

CI for trunk: https://circleci.com/workflow-run/1d30e8e9-8f9e-4b34-be22-e94b7c508561

• Assignee: Zhao Yang
• Reporter: Yakir Gibraltar

Authors

Zhao Yang

Reviewers

Brandon Williams

• Created: 24/Feb/20 13:58
• Updated: 15/May/20 08:54
• Resolved: 27/Apr/20 16:24

Description

Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:

ERROR [ReadStage-6] 2020-02-24 13:53:34,528 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[ReadStage-6,5,main]
java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size: 2761628520
        at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114) ~[apache-cassandra-3.11.6.ja
r:3.11.6]
        at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949) ~[caffeine-2.2.6.jar:na]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807) ~[caffeine-2.2.6.jar:na]
        at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853) ~[na:1.8.0-zing_19.12.102.0]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805) ~[caffeine-2.2.6.jar:na]
        at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788) ~[caffeine-2.2.6.jar:na]
        at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97) ~[caffeine-2.2.6.jar:na]
        at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66) ~[caffeine-2.2.6.jar:na]
        at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.<init>(AbstractSSTableIterator.java:103) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:49) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:423) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0-zing_19.12.102.0]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [apache-cassandra-3.11.6.jar:3.11.6]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) [apache-cassandra-3.11.6.jar:3.11.6]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0-zing_19.12.102.0]

Someone familiar with that error?