[CASSANDRA-10005] Streaming not enough bytes error

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Duplicate
• Labels: triaged
• Severity: Low
• Assignee: Yuki Morishita
• Reporter: Chris Moos

Authors

Yuki Morishita

• Created: 06/Aug/15 16:33
• Updated: 16/Apr/19 09:31
• Resolved: 11/Nov/15 22:56

Description

I'm adding a new node to the cluster and I'm seeing a bunch of the errors below and the node never joins. It looks like a deadlock.

After looking through the code it looks like IncomingFileMessage will tell the session to retry on Exceptions (except IOException) but the CompressedInputStream thread is still running and then the retry happens and the deadlock ensues. It might be best to close the StreamReader (and stop the thread) if an Exception happens before retrying.

I'm not sure why I am getting this error to begin with though, might it have something to do with not being able to upgrade my SSTables after going from 2.1.2 -> 2.2.0?

error: null
-- StackTrace --
java.lang.AssertionError
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.checkUnused(LifecycleTransaction.java:428)
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.split(LifecycleTransaction.java:408)
        at org.apache.cassandra.db.compaction.CompactionManager.parallelAllSSTableOperation(CompactionManager.java:268)
        at org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:373)
        at org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:1524)
        at org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:2521)

Issue Links

• duplicates: CASSANDRA-10012 Deadlock when session streaming is retried after exception