Critical Description
When a zk server is running close to its outstanding requests limit, the server incorrectly throttles the sasl request. This leads to the client waiting for the final sasl packet (session is already established) and deferring all non priming packets till then which also includes the ping packets. The client then waits for the final packet but never gets it and times out saying haven't heard from server. This is fatal for services such as HBase which retry for finite attempts and exit post these attempts.
Issue being that in ZooKeeperServer.processPacket(..) incase of sasl we send the response and incorrectly also call cnxn.incrOutstandingRequests(h), which throttles the connection if we're running over outstandingrequests limit, which results in the server not processing the subsequent packet from the client. Also we donot have any pending request to send for the connection and hence never call enableRecv(). We should return after sending response to the sasl request.
Attachments
Issue Links
- links to
