[CASSANDRA-8665] Cassandra does not start with NPE in ColumnFamilyStore.removeUnfinishedCompactionLeftovers

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce
• Fix Version/s: 2.1.3

Environment

Ubuntu 12.04 | C* 2.1.2 | ruby-driver 1.2

• Severity: Normal
• Assignee: Unassigned
• Reporter: Kishan Karunaratne
• Created: 21/Jan/15 23:02
• Updated: 16/Apr/19 09:31
• Resolved: 19/Mar/15 14:38

Description

During a ruby driver endurance/duration test, the following error occurred:

/mnt/systemlogs/system.log:ERROR [main] 2015-01-17 21:18:25,780 CassandraDaemon.java:482 - Exception encountered during startup
/mnt/systemlogs/system.log-java.lang.NullPointerException: null
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:573) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:249) [main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:465) [main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:554) [main/:na]

Here is the system.log leading up to the error:

INFO  [main] 2015-01-17 21:18:24,581 ColumnFamilyStore.java:278 - Initializing system.peers
INFO  [SSTableBatchOpen:1] 2015-01-17 21:18:24,593 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-169 (10533 bytes)
INFO  [SSTableBatchOpen:1] 2015-01-17 21:18:24,597 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-171 (10572 bytes)
INFO  [SSTableBatchOpen:1] 2015-01-17 21:18:24,598 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/peers-37f71aca7dc2383ba70672528af04d4f/system-peers-ka-170 (10581 bytes)
INFO  [main] 2015-01-17 21:18:24,609 ColumnFamilyStore.java:278 - Initializing system.local
INFO  [SSTableBatchOpen:1] 2015-01-17 21:18:24,613 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-679 (5257 bytes)
INFO  [SSTableBatchOpen:1] 2015-01-17 21:18:24,616 SSTableReader.java:392 - Opening /srv/performance/cass/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-678 (5679 bytes)

Cassandra attempted to restart twice unsuccessfully (this error occurred twice) and then gave up; it seems like a corrupt Data.db file? The endurance test consists of a chaos rhino which randomly rolling restarts a node.

The only other significant error is TombstoneOverwhelmingException and is probably unrelated:

/mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,378 SliceQueryFilter.java:218 - Scanned over 100000 tombstones in system.hints; query aborted (see tombstone_failure_threshold)
/mnt/systemlogs/system.log:ERROR [HintedHandoff:2] 2015-01-17 12:46:32,416 CassandraDaemon.java:170 - Exception in thread Thread[HintedHandoff:2,1,main]
/mnt/systemlogs/system.log-org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:220) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:320) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1885) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1693) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:378) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:337) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:88) ~[main/:na]
/mnt/systemlogs/system.log-	at org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:548) ~[main/:na]
/mnt/systemlogs/system.log-	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_25]
/mnt/systemlogs/system.log-	at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

Issue Links

• is related to: CASSANDRA-8544 Cassandra could not start with NPE in ColumnFamilyStore.removeUnfinishedCompactionLeftovers