Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-14694

RPCProducerIdManager should not wait for a new block

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.6.0
    • None
    • None

    Description

      RPCProducerIdManager initiates an async request to the controller to grab a block of producer IDs and then blocks waiting for a response from the controller.

      This is done in the request handler threads while holding a global lock. This means that if many producers are requesting producer IDs and the controller is slow to respond, many threads can get stuck waiting for the lock.

      This may also be a deadlock concern under the following scenario:

      if the controller has 1 request handler thread (1 chosen for simplicity) and receives an InitProducerId request, it may deadlock.
      basically any time the controller has N InitProducerId requests where N >= # of request handler threads has the potential to deadlock.

      consider this:
      1. the request handler thread tries to handle an InitProducerId request to the controller by forwarding an AllocateProducerIds request.
      2. the request handler thread then waits on the controller response (timed poll on nextProducerIdBlock)
      3. the controller's request handler threads need to pick this request up, and handle it, but the controller's request handler threads are blocked waiting for the forwarded AllocateProducerIds response.

       

      We should not block while waiting for a new block and instead return immediately to free the request handler threads.

      Attachments

        Issue Links

          Activity

            People

              jeffkbkim Jeff Kim
              jeffkbkim Jeff Kim
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: