We have a production cluster with 72 instances spread across 2 DCs. We have a large number ( ~ 40,000 ) of clients hitting this cluster. Client normally connects to 4 cassandra instances. Some event (we think it is a schema change on server side) triggered the client to establish connections to all cassandra instances of local DC. This brought the server to its knees. The client connections failed and client attempted re-connections.
Cassandra should protect itself from such attack from client. Do we have any knobs to control the number of max connections? If not, we need to add that knob.