[CASSANDRA-19508] Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when require_client_auth is set to false

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 4.0.13, 4.1.5, 5.0-beta2, 5.1
• Component/s: Feature/Encryption
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0.0

Source Control Link

https://github.com/apache/cassandra/commit/6a640b0c4e6e3aecca08c749250e2dfec18063ab

Test and Documentation Plan

run CI

• Assignee: Mohammad Aburadeh
• Reporter: Mohammad Aburadeh

Authors

Mohammad Aburadeh

Reviewers

Brandon Williams, Jon Meredith

• Created: 31/Mar/24 22:34
• Updated: 04/Apr/24 16:03
• Resolved: 04/Apr/24 16:03

Description

We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started seeing thousands of msgs "Failed to get peer certificates for peer /x.x.x.x:45796". SSL is enabled but require_client_auth is disabled.  This is causing a huge problem for us because cassandra log files are growing very fast as our connections are short live connections, we open more than 1K connections per second and they stay live for 1-2 seconds. 

DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026 ServerConnection.java:140 - Failed to get peer certificates for peer /172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
        at io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
        at org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
        at org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
        at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
        at org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
        at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
        at org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
        at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
        at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
        at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 

Our SSL config:

client_encryption_options:
  enabled: true
  keystore: /path/to/keystore
  keystore_password: xxxxx
  optional: false
  require_client_auth: false 

 

We should stop throwing this msg when require_client_auth is set to false. Or at least it should be logged in TRACE not DEBUG. 

I'm working on preparing a PR.