Critical Description
We're currently rebalancing the utility cache in the system pool. This can cause starvation, for example in this scenario:
- Start several nodes and a cache with indexing configured.
- Continuously load the system with puts.
- Add one more node. It will join and will start processing put requests immediately.
- For each request, indexing processor will try to get binary metadata and will block on utility cache rebalance future. This will block all threads in the system pool.
- Utility cache rebalancing will be scheduled but will never start, because all threads in system pool are blocked.
I think we should rebalance utility cache synchronously in exchange worker, like we do with the marshaller cache.