[CASSANDRA-7593] Min/max column name collection broken with range tombstones

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1 rc5
• Labels: qa-resolved

Environment

java 1.7

• Severity: Critical
• Assignee: Tom Hobbs
• Reporter: Russ Hatch

Authors

Tom Hobbs

Reviewers

Marcus Eriksson

• Tester: Russ Hatch
• Created: 22/Jul/14 22:58
• Updated: 16/Apr/19 09:31
• Resolved: 01/Aug/14 15:38

Description

I'm seeing two different errors cropping up in the dtest which upgrades a cluster through several versions.

This is the more common error:

ERROR [GossipStage:10] 2014-07-22 13:14:30,028 CassandraDaemon.java:168 - Exception in thread Thread[GossipStage:10,5,main]
java.lang.AssertionError: null
        at org.apache.cassandra.db.filter.SliceQueryFilter.shouldInclude(SliceQueryFilter.java:347) ~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.shouldInclude(QueryFilter.java:249) ~[main/:na]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:249) ~[main/:na]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345) ~[main/:na]
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:293) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:302) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:60) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:263) ~[main/:na]
        at org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:514) ~[main/:na]
        at org.apache.cassandra.net.OutboundTcpConnectionPool.<init>(OutboundTcpConnectionPool.java:51) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:522) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:536) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:689) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:663) ~[main/:na]
        at org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40) ~[main/:na]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[main/:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_60]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]

The same test sometimes fails with this exception instead:

ERROR [CompactionExecutor:4] 2014-07-22 16:18:21,008 CassandraDaemon.java:168 - Exception in thread Thread[CompactionExecutor:4,1,RMI Runtime]
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7059d3e9 rejected from org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@108f1504[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 95]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) ~[na:1.7.0_60]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:325) ~[na:1.7.0_60]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:530) ~[na:1.7.0_60]
        at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:619) ~[na:1.7.0_60]
        at org.apache.cassandra.io.sstable.SSTableReader.scheduleTidy(SSTableReader.java:628) ~[main/:na]
        at org.apache.cassandra.io.sstable.SSTableReader.tidy(SSTableReader.java:609) ~[main/:na]
        at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1596) ~[main/:na]
        at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:354) ~[main/:na]
        at org.apache.cassandra.io.sstable.SSTableRewriter.abort(SSTableRewriter.java:200) ~[main/:na]
        at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:192) ~[main/:na]
        at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[main/:na]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[main/:na]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[main/:na]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[main/:na]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:236) ~[main/:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]