[CASSANDRA-15302] Pool is busy (no available connection and the queue has reached its max size 256)))

• Type: Bug
• Status: Triage Needed
• Priority: Normal
• Resolution: Unresolved
• Component/s: Cluster/Schema
• Platform: All
• Assignee: Unassigned
• Reporter: debasis maity
• Created: 05/Sep/19 17:24
• Updated: 05/Sep/19 17:24

Description

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: 192.168.11.16/192.168.11.16:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [192.168.11.16/192.168.11.16] Pool is busy (no available connection and the queue has reached its max size 256)))
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:83)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
....

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:83)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:37)
at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
.....

We believe this has caused the caching structures in Consolidator to fall out of sync with newly provisioned devices/types, so the above error was provoked.
After clearing the cache by restarting memory cache bundles, the input is WORKING CORRECTLY now.

However, there are still problems related to Cassandra server, as sometimes the output doesn't work (reporting the errors like NoAvailableConnections above) etc.
Please check the status of Cassandra cluster (note that the other components are also using it).

 

nodetool status is showing fine.