The ClientState.login() function is used for all auth message: AuthResponse.java:82. But the role.canLogin information is not cached. So it hits the database every time: CassandraRoleManager.java:407. For a cluster with lots of new connections, it's causing performance issue. The mitigation for us is to increase the system_auth replication factor to match the number of nodes, so local_one would be very cheap. The P99 dropped immediately, but I don't think it is not a good solution.
I would purpose to add Role.canLogin to the RolesCache to improve the auth performance.