Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.10.0
-
None
-
None
Description
We hit a distributed deadlock in one of our tests where two members are hung sending startup messages to each other.
It turns out that until a member gets a response to a StartupMessage, it is in a state where it blocks all outgoing messages. At the same time, the member is receiving an attempting to respond to other messages, but those responses get blocked. If too many messages come in before the StartupResponseMessage, this ends up filling up the ClusterDistributionManager.highPriorityPool.
If two members are trying to start up at the same time, and they both fill up the highPriorityPool, they both will fail to process each other's StartupMessage, because that message is executed in the same pool.
