Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6840

2.0.{5,6} node throws EOFExceptions on the Row mutation forwarding path during rolling upgrade from 1.2.15.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.0.7
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      During a rolling upgrade from 1.2.15 to 2.0.5 nodes running on 2.0.5 throw an EOFException:

      ERROR [MutationStage:12] 2014-03-12 09:46:35,706 RowMutationVerbHandler.java (line 63) Error in row mutation
      java.io.EOFException
      	at java.io.DataInputStream.readFully(DataInputStream.java:197)
      	at org.apache.cassandra.net.CompactEndpointSerializationHelper.deserialize(CompactEndpointSerializationHelper.java:37)
      	at org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNodes(RowMutationVerbHandler.java:81)
      	at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:49)
      	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

      In this specific context we have a setup with 3 datacenters, 3 nodes in each datacenter, NetworkTopologyStrategy as placement_strategy with 3 replicas in each DC. We noticed the issue on the only 2.0.5 node in the ring. All nodes run on Java7. We have tried to upgrade the node on 2.0.5 to 2.0.6 but that didn't solve the issue.

      At a first glance it seems that the size of the size of the list of forward addresses in org.apache.cassandra.db.RowMutationVerbHandler.forwardToLocalNodes() in inconsistent with the length of the InputStream, which causes the deserializer to try and read after the end of the InputStream.

        Attachments

          Activity

            People

            • Assignee:
              marcuse Marcus Eriksson
              Reporter:
              fpiccinini Federico Piccinini
              Authors:
              Marcus Eriksson
              Reviewers:
              Jonathan Ellis
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: