When (for whatever reason) one or more daemons are disconnected from the statestore the admission control data held on the daemon goes stale. This can lead to the daemon accepting queries when there is not capacity or rejecting queries when there is capacity.
For example, a pool somepool has a limit of 10 concurrent queries and is at that limit when a daemon is disconnected from the statestore. Even when other queries in somepool finish and the pool is now empty the disconnected daemon will report the following when new queries are executed:
ERROR: Admission for query exceeded timeout 60000ms. Queued reason: number of running queries 10 is over limit 10
Could we have some warning to say that the admission control data is stale here?