Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14191

Bootstrap/Streaming fails with missing CompressionInfo

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Labels:
      None
    • Severity:
      Normal

      Description

      Multiple attempts at bootstrapping a new node fail, with streaming failing (either hanging or stopping the bootstrap node) always from the same node.

       

      The original node throws the following exception during the streaming process:

      ERROR [STREAM-OUT-/10.83.74.236:47220] 2018-01-24 19:25:22,532 StreamSession.java:512 - [Stream #90c1c8b0-013a-11e8-b5f0-9323de372ca2] Streaming error occurred on session with peer X.X.X.X
      java.lang.AssertionError: null
      	at org.apache.cassandra.io.compress.CompressionMetadata$Chunk.<init>(CompressionMetadata.java:473) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:287) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.serialize(FileMessageHeader.java:172) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:82) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:49) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:41) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:377) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:349) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
      

      The bootstrapping node's reaction to this failure is

      ERROR [STREAM-IN-/10.83.74.234:7001] 2018-01-24 19:25:22,957 StreamSession.java:512 - [Stream #90c1c8b0-013a-11e8-b5f0-9323de372ca2] Streaming error occurred on session with peer X.X.X.X
      java.io.EOFException: null
      	at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[na:1.8.0_151]
      	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.deserialize(CompressionInfo.java:68) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.compress.CompressionInfo$CompressionInfoSerializer.deserialize(CompressionInfo.java:47) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.FileMessageHeader$FileMessageHeaderSerializer.deserialize(FileMessageHeader.java:188) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:42) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:276) ~[cassandra-all-2.1.18.1463.jar:2.1.18.1463]
      	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
      

      Other observations:

      • always the one node that fails,
      • multiple bootstrap attempts (using different ec2 instances) all fail,
      • the exception occurs to -tmp- sstables that have no CompressionInfo component,
      • it's a different -tmp- sstable each time,
      • running either nodetool cleanup or nodetool scrub made no difference,

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              michaelsembwever mck
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: