[CASSANDRA-452] Corrupt SSTable

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.5

Environment

Pre 0.4 based on r805615 on trunk w/ #370, #392, #394, #405, #406, #418

• Severity: Normal
• Assignee: Jonathan Ellis
• Reporter: Sammy Yu

Authors

Jonathan Ellis

• Created: 22/Sep/09 22:06
• Updated: 16/Apr/19 09:33
• Resolved: 10/Nov/09 05:03

Description

We noticed on one of our node the number of SStables is growing. The compaction thread is alive and running. We can see that it is constantly trying to compact the same set of sstables. However, it is failing because one of the sstable is corrupt:

ERROR [ROW-READ-STAGE:475] 2009-09-21 00:29:17,068 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.io.EOFException
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:110)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)
at org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:390)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:64)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:349)
at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:309)
at org.apache.cassandra.db.filter.SSTableNamesIterator.<init>(SSTableNamesIterator.java:102)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1467)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1401)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1420)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1401)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
... 4 more

Sub-Tasks

• skip corrupt rows during compaction instead of erroring out and aborting: Resolved