Details
-
Improvement
-
Status: Open
-
Normal
-
Resolution: Unresolved
Description
In production, there could be hundreds of clients connected to a Cassandra cluster (maybe even from different applications), and if they use DataStax Java Driver, each client will establish at least one TCP connection to a Cassandra server (see https://datastax.github.io/java-driver/2.1.9/features/pooling/). This is all normal and at any given time, you can indeed see hundreds of ESTABLISHED connections to port 9042 on a C* server (from netstat -na). The problem is that sometimes when a C* cluster is under heavy load, when the DBA identifies some client session that sends abusive amount of traffic to the C* server and would like to stop it, they would like a lightweight approach rather than shutting down the JVM or rolling restart the whole cluster to kill all hundreds of connections in order to kill a single client session. If the DBA had root privilege, they would have been able to do something at the OS network level to achieve the same goal but oftentimes enterprise DBA role is separate from OS sysadmin role, so the DBAs usually don't have that privilege.
This is especially helpful when you have a multi-tenant C* cluster and you want to have the impact for handling such client to be minimal to the other applications. This feature (killing individual session) seems to be a common feature in other databases (regardless of whether the client has some reconnect logic or not). It could be implemented as a JMX MBean method and exposed through nodetool to the DBAs.
Note due to CQL driver's automated reconnection, simply killing the currently connected client session will not work well, so the JMX parameter should be an IP address or a list of IP addresses, so that the Cassandra server can terminate existing connection with that IP, and block future connection attempts from that IP for the remaining time until the JVM is restarted.