Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16068

SSL connection to storage port when internode encryption is disabled still try to load keystore

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: None
    • Component/s: Messaging/Internode
    • Labels:
      None
    • Platform:
      All
    • Impacts:
      None

      Description

      Starting a 4.0 cluster with internode encryption disabled can throw an exception if the keystore is not present after CASSANDRA-15262.

      Part of cassandra.yaml, the keystore is optional and it just defaults to conf/.keystore

      server_encryption_options:
          internode_encryption: none
          keystore: conf/.keystore
          keystore_password: cassandra
          truststore: conf/.truststore
          truststore_password: cassandra
      

      Start the service and try to connect with openssl

      $ openssl s_client -connect 127.0.0.1:7000
      CONNECTED(00000003)
      4790519404:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:ssl_pkt.c:585:
      ---
      no peer certificate available
      ---
      No client certificate CA names sent
      ---
      SSL handshake has read 0 bytes and written 0 bytes
      ---
      New, (NONE), Cipher is (NONE)
      Secure Renegotiation IS NOT supported
      Compression: NONE
      Expansion: NONE
      No ALPN negotiated
      SSL-Session:
          Protocol  : TLSv1.2
          Cipher    : 0000
          Session-ID:
          Session-ID-ctx:
          Master-Key:
          Start Time: 1597969961
          Timeout   : 7200 (sec)
          Verify return code: 0 (ok)
      ---
      

      Which triggers an ERROR message with exception

      ERROR [Messaging-EventLoop-3-1] 2020-08-20 18:34:52,855 InboundConnectionInitiator.java:355 - Failed to properly handshake with peer /127.0.0.1:61851. Closing the channel.
      io.netty.handler.codec.DecoderException: java.io.IOException: failed to build trust manager store for secure connections
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
              at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
              at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
              at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.io.IOException: failed to build trust manager store for secure connections
              at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:232)
              at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:300)
              at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:276)
              at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:257)
              at org.apache.cassandra.net.InboundConnectionInitiator$OptionalSslHandler.decode(InboundConnectionInitiator.java:492)
              at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
              ... 17 common frames omitted
      Caused by: java.nio.file.NoSuchFileException: conf/.keystore
              at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
              at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
              at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
              at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
              at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
              at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
              at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
              at java.base/java.nio.file.Files.newInputStream(Files.java:155)
              at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:207)
              ... 23 common frames omitted
      ERROR [Messaging-EventLoop-3-1] 2020-08-20 18:34:52,858 InboundConnectionInitiator.java:355 - Failed to properly handshake with peer /127.0.0.1:61851. Closing the channel.
      io.netty.handler.codec.DecoderException: java.io.IOException: failed to build trust manager store for secure connections
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
              at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404)
              at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371)
              at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
              at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
              at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
              at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818)
              at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
              at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
              at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: java.io.IOException: failed to build trust manager store for secure connections
              at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:232)
              at org.apache.cassandra.security.SSLFactory.createNettySslContext(SSLFactory.java:300)
              at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:276)
              at org.apache.cassandra.security.SSLFactory.getOrCreateSslContext(SSLFactory.java:257)
              at org.apache.cassandra.net.InboundConnectionInitiator$OptionalSslHandler.decode(InboundConnectionInitiator.java:492)
              at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
              at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
              ... 18 common frames omitted
      Caused by: java.nio.file.NoSuchFileException: conf/.keystore
              at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
              at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
              at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
              at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
              at java.base/java.nio.file.Files.newByteChannel(Files.java:370)
              at java.base/java.nio.file.Files.newByteChannel(Files.java:421)
              at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
              at java.base/java.nio.file.Files.newInputStream(Files.java:155)
              at org.apache.cassandra.security.SSLFactory.buildKeyManagerFactory(SSLFactory.java:207)
              ... 24 common frames omitted
      

      If the optional SSL handler is disabled, Cassandra emits this error instead.

      ERROR [Messaging-EventLoop-3-1] 2020-08-20 18:44:34,338 InboundConnectionInitiator.java:358 - Failed to properly handshake with peer /127.0.0.1:61967. Closing the channel.
      io.netty.handler.codec.DecoderException: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 369295616, Expected -900387334
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:404)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:371)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
      	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
      	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:818)
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
      	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      Caused by: org.apache.cassandra.net.Message$InvalidLegacyProtocolMagic: Read 369295616, Expected -900387334
      	at org.apache.cassandra.net.Message.validateLegacyProtocolMagic(Message.java:333)
      	at org.apache.cassandra.net.HandshakeProtocol$Initiate.maybeDecode(HandshakeProtocol.java:167)
      	at org.apache.cassandra.net.InboundConnectionInitiator$Handler.initiate(InboundConnectionInitiator.java:258)
      	at org.apache.cassandra.net.InboundConnectionInitiator$Handler.decode(InboundConnectionInitiator.java:251)
      	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440)
      	... 18 common frames omitted
      

      The system should either not register the SSL handler when encryption is disabled (the behavior before CASSANDRA-15262) or gracefully tolerate the error.

      Additionally, the inbound connection should either be an INFO or a WARNING. Outbound connection failures should be an ERROR as the endpoint is expected, however inbound connections have to deal with anything connecting - from security scans to malicious actors

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jmeredithco Jon Meredith
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: