[CASSANDRA-12752] Exception encountered during startup (CFMetaData.rebuild failed)

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Information Provided

Environment

Cassandra 3.0.7, Debian Jessie. Java version 64 bits : 1.8.0_91-b14, SSD software raid 1

• Severity: Normal
• Assignee: Unassigned
• Reporter: Jérôme Schneider
• Created: 06/Oct/16 08:20
• Updated: 16/Apr/19 09:30
• Resolved: 19/Nov/18 02:08

Description

We have a cluster of 3 cassandra 3.0.7. We have the following problems :

• ssltables was to big to be compress (not enough space left)
• we launch a sstablesplit which seems to work
• after we can't restart cassandra nodes we always have the same stacktrace during the start :

  INFO [main] 2016-10-05 18:02:47,346 ColumnFamilyStore.java:390 - Initializing system_schema.aggregates
  INFO [main] 2016-10-05 18:02:47,353 ColumnFamilyStore.java:390 - Initializing system_schema.indexes
  ERROR [main] 2016-10-05 18:02:47,420 CassandraDaemon.java:698 - Exception encountered during startup
  java.lang.AssertionError: null
  at org.apache.cassandra.db.marshal.CompositeType.getInstance(CompositeType.java:103) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.config.CFMetaData.rebuild(CFMetaData.java:313) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.config.CFMetaData.<init>(CFMetaData.java:292) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.config.CFMetaData.create(CFMetaData.java:368) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:954) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:928) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:891) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:868) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:856) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:136) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:126) ~[apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:235) [apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [apache-cassandra-3.0.7.jar:3.0.7]
  at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685) [apache-cassandra-3.0.7.jar:3.0.7]

• it's now impossible to start a cassandra node. I tried to : upgrade to 3.0.9 (same error), I reset system_schema (cassandra start) and I tried to launch sstablescrub on the corrupted keystore (same stacktrace).