Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Not A Problem
-
None
-
None
-
None
-
All
-
None
Description
I am facing issue with my cassandra cluster, we are observing connection timeout errors with BusyPoolException.
Logs:
java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /x.x.x.x:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [/x.x.x.x:9042] Timed out waiting for server response), cassandra/x.x.x.x:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [cassandra/x.x.x.x:9042] Pool is busy (no available connection and the queue has reached its max size 0)))java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /x.x.x.x:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [/x.x.x.x:9042] Timed out waiting for server response), cassandra/x.x.x.x:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [cassandra/x.x.x.x:9042] Pool is busy (no available connection and the queue has reached its max size 0))) at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:513) at akka.persistence.cassandra.package$ListenableFutureConverter$$anon$2.$anonfun$run$2(package.scala:50) at scala.util.Try$.apply(Try.scala:213) at akka.persistence.cassandra.package$ListenableFutureConverter$$anon$2.run(package.scala:50) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:47) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:47) at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source) at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /x.x.x.x:9042 (com.datastax.driver.core.exceptions.OperationTimedOutException: [/x.x.x.x:9042] Timed out waiting for server response), cassandra/x.x.x.x:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [cassandra/x.x.x.x:9042] Pool is busy (no available connection and the queue has reached its max size 0))) at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:283) at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:61) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:375) at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:444) at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1015) at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1137) at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:707) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.addListener(AbstractFuture.java:112) at com.google.common.util.concurrent.Futures.addCallback(Futures.java:996) at com.datastax.driver.core.GuavaCompatibility.addCallback(GuavaCompatibility.java:112) at com.datastax.driver.core.GuavaCompatibility.addCallback(GuavaCompatibility.java:100) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:400) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:359) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:557) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:539) at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onTimeout(RequestHandler.java:981) at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1605) at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:680) at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:755) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:483) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Unknown Source)