[CASSANDRA-7275] Errors in FlushRunnable may leave threads hung

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Won't Fix
• Severity: Low
• Assignee: Unassigned
• Reporter: Tom Hobbs
• Created: 21/May/14 00:14
• Updated: 16/Apr/19 09:31
• Resolved: 24/Aug/15 16:23

Description

In Memtable.FlushRunnable, the CountDownLatch will never be counted down if there are errors, which results in hanging any threads that are waiting for the flush to complete. For example, an error like this causes the problem:

ERROR [FlushWriter:474] 2014-05-20 12:10:31,137 CassandraDaemon.java (line 198) Exception in thread Thread[FlushWriter:474,5,main]
java.lang.IllegalArgumentException
    at java.nio.Buffer.position(Unknown Source)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:64)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:72)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.split(AbstractCompositeType.java:138)
    at org.apache.cassandra.io.sstable.ColumnNameHelper.minComponents(ColumnNameHelper.java:103)
    at org.apache.cassandra.db.ColumnFamily.getColumnStats(ColumnFamily.java:439)
    at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:194)
    at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:397)
    at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350)
    at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Issue Links

• is duplicated by: CASSANDRA-8476 RE in writeSortedContents or replaceFlushed blocks compaction threads indefinitely.
• is duplicated by: CASSANDRA-9255 If a memtable flush fails, possible deadlock for future flushes
• is related to: CASSANDRA-7373 Commit logs no longer deleting and MemtablePostFlusher pending growing
• relates to: CASSANDRA-8496 Remove MemtablePostFlusher
• relates to: CASSANDRA-8498 Replaying commit log records that are older than gc_grace is dangerous
• relates to: CASSANDRA-8497 Do not replay commit log records for tables that have been repaired since