[CASSANDRA-18541] AUTH requests use too much resources

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Platform: All
• Assignee: Unassigned
• Reporter: Yury Vidineev
• Created: 20/May/23 07:17
• Updated: 27/Jun/23 23:20
• Resolved: 27/Jun/23 23:20

Description

Hello. I see unexpected CPU usage in a rare situation that may be worth digging into.
We have C* 4.0.9 on Debian running on Java 11.0.18.
It's a small cluster of 3 nodes on commodity hardware (6 cores CPU, 32 Gb RAM, 2 x 512 Gb SSD NVME).
This ring has about 35 clients using Datastax Java Driver for Apache Cassandra.
In the driver connection settings, we use the following:
CONNECTION_POOL_LOCAL_SIZE = 400
CONNECTION_POOL_REMOTE_SIZE = 100
 
And for some reason, from time to time, it causes hundreds of AUTH requests per second that leads to an enormous CPU usage.
And yes, it's easy not to use these settings in the driver, leaving defaults that don't produce such an amount of AUTHs. But isn't it weird that ~150 AUTH rps consume ~1200% CPU?
Please see attached graphs.

I have the following in the settings:

authenticator: PasswordAuthenticator

authorizer: CassandraAuthorizer

roles_validity_in_ms: 600000

permissions_validity_in_ms: 600000

credentials_validity_in_ms: 600000

Please let me know if I can provide any other necessary information.
Thanks for your work. Cassandra is amazing

Issue Links

• is related to: CASSANDRA-17812 Rate-limit new client connection setup to avoid overwhelming bcrypt