Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9359

Controller does not handle requests while broker is being shutdown

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: controller, core
    • Labels:
      None

      Description

      When a broker is shutdown it first tries to go through a controlled shutdown, resigning leadership of its partitions and then it stops the socket server from processing requests and shuts down the various data plane and control plane handlers and processors:

       

      if (socketServer != null)
        CoreUtils.swallow(socketServer.stopProcessingRequests(), this)
      if (dataPlaneRequestHandlerPool != null)
        CoreUtils.swallow(dataPlaneRequestHandlerPool.shutdown(), this)
      if (controlPlaneRequestHandlerPool != null)
        CoreUtils.swallow(controlPlaneRequestHandlerPool.shutdown(), this)
      if (kafkaScheduler != null)
        CoreUtils.swallow(kafkaScheduler.shutdown(), this)
      
      if (dataPlaneRequestProcessor != null)
        CoreUtils.swallow(dataPlaneRequestProcessor.close(), this)
      if (controlPlaneRequestProcessor != null)
        CoreUtils.swallow(controlPlaneRequestProcessor.close(), this)

      The kafkaController component is only shut down much later, after closing the logManager, a process which may take some time as log closing requires checkpointing state and flushing segments. If the broker being shutdown is the controller, this means there is a potentially large window in which no controller is processing controller requests. Only when the controller component is shutdown and the zkClient is closed will the controller resign leadership.

      There is a second problem in that a broker that does not successfully undergo controlled shutdown will also remain the leader for its partitions until the zkClient is shutdown, and the potential window there is large due to the aforementioned log manager shutdown.

      It would be ideal if:

      1. controller leadership is resigned early in the shutdown process before request handling is stopped. Care will have to be taken so that the broker in question cannot regain it.
      2. we can reduce the window between an uncontrolled shutdown and resigning leadership of partitions through the zkclient close failsafe.

      See also https://issues.apache.org/jira/browse/KAFKA-9358

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lucasbradstreet Lucas Bradstreet
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: