Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10536

Batch statements with multiple updates to partition error when table is indexed

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.0.0 rc2
    • Component/s: Legacy/CQL
    • Labels:
      None
    • Severity:
      Normal

      Description

      If a BATCH statement contains multiple UPDATE statements that update the same partition, and a secondary index exists on that table, the batch statement will error:

      ServerError: <ErrorMessage code=0000 [Server error] message="java.lang.IllegalStateException: An update should not be written again once it has been read">
      

      with the following traceback in the logs:

      ERROR 20:53:46 Unexpected exception during request
      java.lang.IllegalStateException: An update should not be written again once it has been read
      	at org.apache.cassandra.db.partitions.PartitionUpdate.assertNotBuilt(PartitionUpdate.java:504) ~[main/:na]
      	at org.apache.cassandra.db.partitions.PartitionUpdate.add(PartitionUpdate.java:535) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:96) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:667) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.getMutations(BatchStatement.java:234) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:335) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:321) ~[main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:316) ~[main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:205) ~[main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:471) ~[main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:448) ~[main/:na]
      	at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) ~[main/:na]
      	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
      	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
      	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
      	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na]
      	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
      

      This is due to SecondaryIndexManager.validate() triggering a build of the PartitionUpdate (stacktrace from debugging the build() call):

      at org.apache.cassandra.db.partitions.PartitionUpdate.build(PartitionUpdate.java:571) [main/:na]
      	at org.apache.cassandra.db.partitions.PartitionUpdate.maybeBuild(PartitionUpdate.java:561) [main/:na]
      	at org.apache.cassandra.db.partitions.PartitionUpdate.iterator(PartitionUpdate.java:418) [main/:na]
      	at org.apache.cassandra.index.internal.CassandraIndex.validateRows(CassandraIndex.java:560) [main/:na]
      	at org.apache.cassandra.index.internal.CassandraIndex.validate(CassandraIndex.java:314) [main/:na]
      	at org.apache.cassandra.index.SecondaryIndexManager.lambda$validate$75(SecondaryIndexManager.java:642) [main/:na]
      	at org.apache.cassandra.index.SecondaryIndexManager$$Lambda$166/1388080038.accept(Unknown Source) [main/:na]
      	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) [na:1.8.0_45]
      	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) [na:1.8.0_45]
      	at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566) [na:1.8.0_45]
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) [na:1.8.0_45]
      	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) [na:1.8.0_45]
      	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) [na:1.8.0_45]
      	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) [na:1.8.0_45]
      	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [na:1.8.0_45]
      	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) [na:1.8.0_45]
      	at org.apache.cassandra.index.SecondaryIndexManager.validate(SecondaryIndexManager.java:642) [main/:na]
      	at org.apache.cassandra.cql3.UpdateParameters.validateIndexedColumns(UpdateParameters.java:105) [main/:na]
      	at org.apache.cassandra.cql3.statements.UpdateStatement.addUpdateForKey(UpdateStatement.java:107) [main/:na]
      	at org.apache.cassandra.cql3.statements.ModificationStatement.addUpdates(ModificationStatement.java:667) [main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.getMutations(BatchStatement.java:234) [main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:335) [main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:321) [main/:na]
      	at org.apache.cassandra.cql3.statements.BatchStatement.execute(BatchStatement.java:316) [main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:205) [main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:471) [main/:na]
      	at org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:448) [main/:na]
      	at org.apache.cassandra.transport.messages.ExecuteMessage.execute(ExecuteMessage.java:130) [main/:na]
      	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [main/:na]
      	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [main/:na]
      	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45]
      	at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [main/:na]
      	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [main/:na]
      	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
      

      You can reproduce the issue with a table and index like this:

      CREATE TABLE foo (a int, b int, c int, d int, PRIMARY KEY (a, b));
      CREATE INDEX ON foo(c);
      

      and a batch update like this:

      BEGIN BATCH
      UPDATE foo SET c = 0 WHERE a = 0 AND b = 0;
      UPDATE foo SET d = 0 WHERE a = 0 AND b = 0;
      APPLY BATCH
      

      Assigning Sylvain Lebresne for now, since he probably has the best idea of how to avoid this.

        Attachments

          Activity

            People

            • Assignee:
              samt Sam Tunnicliffe
              Reporter:
              thobbs Tom Hobbs
              Authors:
              Sam Tunnicliffe
              Reviewers:
              Aleksey Yeschenko
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: