Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14839

Cannot bootstrap new node IOException: CF <GUID> was dropped during streaming - Cassandra 3.0.16

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Cassandra 3.0.16

    • Severity:
      Normal

      Description

      I have a Cassandra 3.0.16 cluster with 6 nodes. 

      Repairs work well on all nodes. 

      Trying to join a 7th node fails with:

      ERROR [STREAM-IN-/172.30.2.191] 2018-10-23 20:57:06,497 StreamSession.java:534 - [Stream #08d98990-d706-11e8-bf10-9b02b0f15309] Streaming error occurred
      java.io.IOException: CF 9428b300-99ac-11e8-aa56-b1d7e983933e was dropped during streaming
      at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:54) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:43) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:294) ~[apache-cassandra-3.0.16.jar:3.0.16]
      at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.0.16.jar:3.0.16]
      at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]
      

      On disk I have:

      ubuntu@prd-cassandra-2d:/data$ find -type d -name '*9428b30099ac11e8aa56b1d7e983933e*'
      ./cassandra/data/sandboxprdvault/vault-9428b30099ac11e8aa56b1d7e983933e
      

      However in system_schema.tables I have:

      keyspace_name	table_name	id
      sandboxprdvault	vault	942c8390-99ac-11e8-be55-69bc3438b9fa
      

      What is unusual to me is that the system_schema.table.ID for vault is different than the id on disk for the same table. 

      This seems to cause a stream error when bootstrapping a new node. 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              scubadrew Drew O'Connor
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: