Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Not A Problem
-
None
-
None
-
None
-
RHEL 6.5
Cassandra 1.2.16
RF=3
Thrift
-
Normal
Description
I've found several unresolved JIRA tickets related to SSTable corruption but not sure if they apply to the case we are seeing in system/hints. We see periodic exceptions such as:
dataSize of 144115248479299639 starting at 17209 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
Is there something we could possibly be doing from the application to cause this sort of corruption? We also see it on some of our own column families also some negative lengths which are presumably a similar corruption.
ERROR [HintedHandoff:57] 2014-06-17 17:08:04,690 CassandraDaemon.java (line 191) Exception in thread Thread[HintedHandoff:57,1,main]
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 144115248479299639 starting at 17209 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:441)
at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:282)
at org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:90)
at org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:508)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 144115248479299639 starting at 17209 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:437)
... 6 more
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.IOException: dataSize of 144115248479299639 starting at 17209 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:167)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:83)
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:69)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:180)
at org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.next(SSTableScanner.java:155)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:142)
at org.apache.cassandra.io.sstable.SSTableScanner.next(SSTableScanner.java:38)
at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:145)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:122)
at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:96)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at org.apache.cassandra.db.compaction.CompactionManager$7.runMayThrow(CompactionManager.java:442)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
Caused by: java.io.IOException: dataSize of 144115248479299639 starting at 17209 would be larger than file /home/y/var/cassandra/data/system/hints/system-hints-ic-219-Data.db length 35542
at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:123)
... 23 more