Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7144

Fix handling of empty counter replication mutations

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 1.2.17, 2.0.9
    • Component/s: None
    • Labels:
      None
    • Environment:

      Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / 2 Cores in DigitalOcean.

      Description

      First time reporting a bug here, apologies if I'm not posting it in the right space.

      At what seems like random interval, on random nodes in random situations I will get the following exception. After this the hinted handoff start timing out and the node stops participating in the cluster.

      I started seeing these after switching to the Cassandra Python-Driver from the Python-CQL driver.

      ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java (line 198) Exception in thread Thread[WRITE-/10.128.180.108,5,main]
      java.lang.AssertionError
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
      	at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
      	at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
      	at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
      	at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
      ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java (line 198) Exception in thread Thread[WRITE-/10.128.194.70,5,main]
      java.lang.AssertionError
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
      	at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
      	at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
      	at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
      	at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
      ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line 198) Exception in thread Thread[MutationStage:118,5,main]
      java.lang.AssertionError
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
      	at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
      	at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
      	at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
      	at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
      	at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line 198) Exception in thread Thread[MutationStage:117,5,main]
      java.lang.AssertionError
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
      	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
      	at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
      	at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
      	at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
      	at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
      	at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

      The service must be restarted for the node to come back online. Let me know any additional configuration details needed.

        Attachments

        1. 7144.txt
          2 kB
          Aleksey Yeschenko
        2. 7144-2.txt
          2 kB
          Aleksey Yeschenko

          Activity

            People

            • Assignee:
              iamaleksey Aleksey Yeschenko
              Reporter:
              nesnub Maxime Lamothe-Brassard
              Reviewer:
              Richard Low
            • Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: