Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15395

ConsistencyLevel randomly changes to ALL

Agile BoardAttach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Triage Needed
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Consistency/Repair
    • Labels:
      None
    • Platform:
      All
    • Impacts:
      None

      Description

      Versions

      Cassandra

      3.11.4

      Datastax Core Driver Java

      3.6

       

      In our config we set the consistency level globally in the cluster

      lazy val session = Cluster
        .builder()
        .addContactPoints(hosts: _*)
        .withPort(port)
        .withQueryOptions(new QueryOptions().setConsistencyLevel(ConsistencyLevel.LOCAL_QUORUM))
        .withCodecRegistry(CodecRegistry.DEFAULT_INSTANCE.register(CassandraTypeConverter.ReviewStatusCodec))
        .build
        .connect
      

      But from time to time we get an error in our logs that is as follows

      com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency ALL (3 responses were required but only 2 replica responded)
      	at com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:124)
      	at com.datastax.driver.core.Responses$Error.asException(Responses.java:169)
      	at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:627)
      	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
      	at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
      	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:546)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:500)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      

       Which states that a read with CL -> ALL has been made.

       

      This old issue https://issues.apache.org/jira/browse/CASSANDRA-7868 had the same problem but it says it was resolved in 3.0.13. I currently have a higher version than that. I'm not sure if the bug is back or this is caused by something else.

        Attachments

          Activity

            People

              Dates

              • Created:
                Updated:

                Issue deployment