Thread [Sender-/127.0.0.1:3084] (Running)
When I observed QuorumZxidSyncTest, the above thread is also leaking in a different scenario apart from ReadOnlyZKServer. Above mentioned thread will be shutdown only if it receives proposalOfDeath.
We are queuing proposalOfDeath in finally block of LearnerHandler.run().
To summarize, proposalOfDeath may not queued up when LearnerHandler receives IOException and other thread calling LearnerHandler.shutdown(). This leads to failure of queuing the proposalOfDeath.
To solve this, can we add the proposalOfDeath in shutdown() rather in finally block.
Anyways, this finding will solve one of the leaks. We still need to fix other leaks caused by ROZK.
Just to see if I understand now, are you saying that the test is spawning over one hundred servers because we are shutting before the server actually starts? If so, then it is certainly a problem.
Yes. I will add more analysis and possibly patch today.