[CASSANDRA-14678] Propose reducing the default value for PasswordAuthenticator number of hashing rounds

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Won't Fix
• Component/s: Feature/Authorization
• Assignee: Unassigned
• Reporter: Shichao An
• Created: 30/Aug/18 01:35
• Updated: 16/Apr/19 09:29
• Resolved: 30/Aug/18 13:29

Description

We saw performance degradation in some of our Cassandra clusters using PasswordAuthenticator. When the clients start connecting to the Cassandra nodes, the CPU load increases, and there is a high chance that the host will be unable to recover from high CPU usage if the clients retry indefinitely at relatively high frequency. In each reconnection, the clients try to initiate auth handshakes, but may fail due to timeouts from the overloaded host, whereas the sporadic auth handshakes will put more load to the host, so on so forth. In our case, the load average can be 1000~3000 on a 32-core host. The host is basically unable to serve any traffic.

We found it is caused by the slow `BCrypt.checkpw` operation, where the generated salted hash round is 10 because `GENSALT_LOG2_ROUNDS_PROPERTY` defaults 10, which makes it 2^10 rounds of hashing iterations. I changed the hashing rounds to 4 by overriding `auth_bcrypt_gensalt_log2_rounds` system property and it can effectively solve above-mentioned the CPU issue.

It took us some time to nail down the cause of this problem. Shall we reduce the default value of `GENSALT_LOG2_ROUNDS_PROPERTY` to a smaller value than 10? Any suggestions on the tradeoff between performance and cryptographic impact?

 

Issue Links

• relates to: CASSANDRA-8085 Make PasswordAuthenticator number of hashing rounds configurable