[CASSANDRA-8851] Uncaught exception on thread Thread[SharedPool-Worker-16,5,main] after upgrade to 2.1.3

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1.5

Environment

ubuntu

• Severity: Critical
• Since Version: 2.1.3
• Assignee: Benedict Elliott Smith
• Reporter: Tobias Schlottke

Authors

Benedict Elliott Smith

• Created: 23/Feb/15 09:23
• Updated: 16/Apr/19 09:31
• Resolved: 19/Mar/15 20:17

Description

Hi there,

after upgrading to 2.1.3 we've got the following error every few seconds:

WARN  [SharedPool-Worker-16] 2015-02-23 10:20:36,392 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-16,5,main]: {}
java.lang.AssertionError: null
	at org.apache.cassandra.io.util.Memory.size(Memory.java:307) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.utils.obs.OffHeapBitSet.capacity(OffHeapBitSet.java:61) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.utils.BloomFilter.indexes(BloomFilter.java:74) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.utils.BloomFilter.isPresent(BloomFilter.java:98) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1366) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.io.sstable.SSTableReader.getPosition(SSTableReader.java:1350) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:41) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:185) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:273) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1915) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1748) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:342) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:57) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45]
	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.3.jar:2.1.3]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3]
	at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

This seems to crash the compactions and pushes up server load and piles up compactions.

Any idea / possible workaround?

Best,

Tobias