[CASSANDRA-2076] Not restarting due to Invalid saved cache

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 0.7.1

Environment

linux

• Severity: Critical
• Assignee: Matthew F. Dennis
• Reporter: Thibaut

Authors

Matthew F. Dennis

Reviewers

Jonathan Ellis

• Created: 31/Jan/11 09:15
• Updated: 16/Apr/19 09:33
• Resolved: 10/Feb/11 03:29

Description

This occured on two nodes on me (running 0.7.1 from svn)

One node was killed by the kernel due to a OOM and the other node was haning and I had to kill it manually with kill -9 (kill didn't work). (maybe these were faulty hardware nodes, I don't know)

The saved_cache was corrupt afterwards and I couldn't start the nodes.

After deleting the saved_caches directory I could start the nodes again.

Instead of not starting when an error occurs, cassandra could simply delete the errornous file and continue to start?

INFO 22:31:11,570 reading saved cache
/hd1/cassandra_md5/saved_caches/table_attributes-table_attributes-KeyCache
ERROR 22:31:11,595 Exception encountered during startup.
java.lang.RuntimeException: The provided key was not UTF8 encoded.
at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:159)
at org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:44)
at org.apache.cassandra.db.ColumnFamilyStore.readSavedCache(ColumnFamilyStore.java:281)
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:218)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:458)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:440)
at org.apache.cassandra.db.Table.initCf(Table.java:360)
at org.apache.cassandra.db.Table.<init>(Table.java:290)
at org.apache.cassandra.db.Table.open(Table.java:107)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:312)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:81)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
at org.apache.cassandra.utils.FBUtilities.decodeToUTF8(FBUtilities.java:403)
at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:155)
... 11 more
Exception encountered during startup.
java.lang.RuntimeException: The provided key was not UTF8 encoded.
at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:159)
at org.apache.cassandra.dht.OrderPreservingPartitioner.decorateKey(OrderPreservingPartitioner.java:44)
at org.apache.cassandra.db.ColumnFamilyStore.readSavedCache(ColumnFamilyStore.java:281)
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:218)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:458)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:440)
at org.apache.cassandra.db.Table.initCf(Table.java:360)
at org.apache.cassandra.db.Table.<init>(Table.java:290)
at org.apache.cassandra.db.Table.open(Table.java:107)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:167)
at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:312)
at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:81)
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:260)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:781)
at org.apache.cassandra.utils.FBUtilities.decodeToUTF8(FBUtilities.java:403)
at org.apache.cassandra.dht.OrderPreservingPartitioner.getToken(OrderPreservingPartitioner.java:155)
... 11 more