Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16524

Upgrading SSL enabled Cassandra cluster from 3.11.10 to 4.0-beta4 failing with javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Hi,

      We have SSL enabled cluster running on Apache Cassandra 3.11.10 and we are trying to upgrade it to 4.0-beta4 as a part of testing.

      Cluster size is 3x3 and deployed on Azure IaaS.

      [cassandra@cass-521828978-1-1189299202 ~]$ nodetool status
      Datacenter: southcentral
      ========================
      Status=Up/Down
      |/ State=Normal/Leaving/Joining/Moving
      --  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
      UN  10.12.74.31  85.61 KiB  16           32.2%             6db7a7ef-3490-4823-9ff3-c60a32165124  2
      UN  10.12.74.42  263.27 KiB  16           27.6%             7ad99ecf-7c7d-4780-872b-7c68b6b19849  1
      UN  10.12.74.34  85.61 KiB  16           37.8%             41ce16b7-2ab2-44ea-a810-8391f7f3caf2  0
      Datacenter: westus
      ==================
      Status=Up/Down
      |/ State=Normal/Leaving/Joining/Moving
      --  Address      Load       Tokens       Owns (effective)  Host ID                               Rack
      UN  10.12.90.11  90.63 KiB  16           38.9%             8d4cdb65-ff66-4bcd-8d4b-a4a0e893a728  2
      UN  10.12.90.6   85.61 KiB  16           34.5%             4f8007e9-fa3e-4e99-a9f9-f99997bf9625  1
      UN  10.12.89.80  94.1 KiB   16           28.9%             11f86cb0-c86b-440e-848f-b160118f43d5  0
      

      We placed a new 4.0-beta4 binary on the first seed node (10.12.74.310) and starting Cassandra.

      It started throwing the below error:

      ERROR [Messaging-EventLoop-3-11] 2021-03-15 22:10:05,188 InboundConnectionInitiator.java:342 - Failed to properly handshake with peer /10.12.74.42:52356. Closing the channel.
      io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException: writerIndex(8560) + minWritableBytes(1977) exceeds maxCapacity(10240): BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
      	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
      	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: javax.net.ssl.SSLException: java.lang.IndexOutOfBoundsException: writerIndex(8560) + minWritableBytes(1977) exceeds maxCapacity(10240): BufferPoolAllocator$Wrapped(ridx: 0, widx: 8560, cap: 10240/10240)
      	at io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterial(OpenSslKeyMaterialManager.java:115)
      	at io.netty.handler.ssl.OpenSslKeyMaterialManager.setKeyMaterialServerSide(OpenSslKeyMaterialManager.java:84)
      	at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext$OpenSslServerCertificateCallback.handle(ReferenceCountedOpenSslServerContext.java:229)
      	at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:596)
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1203)
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1325)
      	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1368)
      	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:206)
      	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1387)
      	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1294)
      	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1331)
      	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
      	... 15 common frames omitted
      

       I have also used the below parameter under server_encryption_options as suggested at : https://cassandra.apache.org/doc/latest/configuration/cass_yaml_file.html#server-encryption-options but still getting the same error.

      enable_legacy_ssl_storage_port: true
      

       
      I am attaching the system.log file here for your review.

      It is working fine with Cassandra 3.11.10 and it looks like some bug in 4.0-beta4.

      Let me know if you need any more details.

      Thanks,
      Alaykumar Barochia

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gianluca Gianluca Righetto Assign to me
            abarochia Alaykumar Barochia
            Gianluca Righetto
            Berenguer Blasi, Ekaterina Dimitrova, Zhao Yang
            Votes:
            0 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment