Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-2785

Server inappropriately throttles connections under load before SASL completes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.4.10
    • Fix Version/s: 3.4.11, 3.5.4, 3.6.0
    • Component/s: server
    • Labels:

      Description

      When a zk server is running close to its outstanding requests limit, the server incorrectly throttles the sasl request. This leads to the client waiting for the final sasl packet (session is already established) and deferring all non priming packets till then which also includes the ping packets. The client then waits for the final packet but never gets it and times out saying haven't heard from server. This is fatal for services such as HBase which retry for finite attempts and exit post these attempts.

      Issue being that in ZooKeeperServer.processPacket(..) incase of sasl we send the response and incorrectly also call cnxn.incrOutstandingRequests(h), which throttles the connection if we're running over outstandingrequests limit, which results in the server not processing the subsequent packet from the client. Also we donot have any pending request to send for the connection and hence never call enableRecv(). We should return after sending response to the sasl request.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                abhishek.chouhan Abhishek Singh Chouhan
                Reporter:
                abhishek.chouhan Abhishek Singh Chouhan
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: