[CASSANDRA-13752] Corrupted SSTables created in 3.11

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate
• Fix Version/s: 3.11.1
• Severity: Critical
• Since Version: 3.11.0
• Assignee: Hannu Kröger
• Reporter: Hannu Kröger

Authors

Hannu Kröger

Reviewers

Jeff Jirsa

• Created: 10/Aug/17 10:03
• Updated: 16/Apr/19 09:30
• Resolved: 29/Aug/17 19:03

Description

We have discovered issues with corrupted SSTables.

ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 - Cannot read sstable /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db, Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt, Index.db, Filter.db]; other IO error, skipping table
java.io.EOFException: EOF after 1898 bytes out of 21093
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561) ~[apache-cassandra-3.11.0.jar:3.11.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.0.jar:3.11.0]

Files look like this:

-rw-r--r--. 1 cassandra cassandra     3899251 Aug  7 08:37 mc-6166-big-CompressionInfo.db
-rw-r--r--. 1 cassandra cassandra 16874421686 Aug  7 08:37 mc-6166-big-Data.db
-rw-r--r--. 1 cassandra cassandra          10 Aug  7 08:37 mc-6166-big-Digest.crc32
-rw-r--r--. 1 cassandra cassandra     2930904 Aug  7 08:37 mc-6166-big-Filter.db
-rw-r--r--. 1 cassandra cassandra   111175880 Aug  7 08:37 mc-6166-big-Index.db
-rw-r--r--. 1 cassandra cassandra       13762 Aug  7 08:37 mc-6166-big-Statistics.db
-rw-r--r--. 1 cassandra cassandra      882008 Aug  7 08:37 mc-6166-big-Summary.db
-rw-r--r--. 1 cassandra cassandra          92 Aug  7 08:37 mc-6166-big-TOC.txt

Issue Links

• duplicates: CASSANDRA-13756 StreamingHistogram is not thread safe
• is broken by: CASSANDRA-6916 Preemptive opening of compaction result
• is related to: CASSANDRA-13038 33% of compaction time spent in StreamingHistogram.update()
• relates to: CASSANDRA-13756 StreamingHistogram is not thread safe