[CASSANDRA-13051] SSTable Corruption - Partition Key fails assertion

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Legacy/Observability

Environment

Cassandra 3.7
Single DC
5 Nodes
RF 3
NetworkTopologyStrategy
OS: Ubuntu

• Severity: Normal
• Since Version: 3.7
• Assignee: Unassigned
• Reporter: Malte Pickhan
• Created: 16/Dec/16 13:21
• Updated: 16/Apr/19 09:30

Description

When running a repair the following exception is triggered:

java.lang.AssertionError: null  
at org.apache.cassandra.db.marshal.TimeUUIDType.compareCustom(TimeUUIDType.java:65) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.marshal.AbstractType.compare(AbstractType.java:157) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:139) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.dht.LocalPartitioner$LocalToken.compareTo(LocalPartitioner.java:120) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:85) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:39) ~[apache-cassandra-3.7.jar:3.7]  
at java.util.concurrent.ConcurrentSkipListMap.cpr(ConcurrentSkipListMap.java:655) ~[na:1.8.0_91]  
at java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:794) ~[na:1.8.0_91]  
at java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1546) ~[na:1.8.0_91]  
at org.apache.cassandra.db.Memtable.getPartition(Memtable.java:355) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionController.maxPurgeableTimestamp(CompactionController.java:221) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionIterator$Purger.getMaxPurgeableTimestamp(CompactionIterator.java:304) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.partitions.PurgeFunction.lambda$new$0(PurgeFunction.java:38) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.DeletionPurger.shouldPurge(DeletionPurger.java:33) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.rows.BTreeRow.purge(BTreeRow.java:386) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.partitions.PurgeFunction.applyToRow(PurgeFunction.java:88) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:120) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.7.jar:3.7]  
at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264) ~[apache-cassandra-3.7.jar:3.7]  
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_91]  
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_91]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_91]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_91]  
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]  

One thing which would be nice is, to provide an actual message in the assertion in order to avoid this "null" string. Furthermore it would be great to provide the data which caused the assertion to fail.

Actually I have no clue, how we triggered this, but I will see if I can find out something...

The only similar Issue I found in the tracker was CASSANDRA-11074, but I am not sure if this is really related?

Issue Links

• is duplicated by: CASSANDRA-13335 AssertionError in PerDiskMemtableFlushWrite when comparing TimeUUIDType