Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When a broker is shutdown it first tries to go through a controlled shutdown, resigning leadership of its partitions and then it stops the socket server from processing requests and shuts down the various data plane and control plane handlers and processors:
if (socketServer != null) CoreUtils.swallow(socketServer.stopProcessingRequests(), this) if (dataPlaneRequestHandlerPool != null) CoreUtils.swallow(dataPlaneRequestHandlerPool.shutdown(), this) if (controlPlaneRequestHandlerPool != null) CoreUtils.swallow(controlPlaneRequestHandlerPool.shutdown(), this) if (kafkaScheduler != null) CoreUtils.swallow(kafkaScheduler.shutdown(), this) if (dataPlaneRequestProcessor != null) CoreUtils.swallow(dataPlaneRequestProcessor.close(), this) if (controlPlaneRequestProcessor != null) CoreUtils.swallow(controlPlaneRequestProcessor.close(), this)
The kafkaController component is only shut down much later, after closing the logManager, a process which may take some time as log closing requires checkpointing state and flushing segments. If the broker being shutdown is the controller, this means there is a potentially large window in which no controller is processing controller requests. Only when the controller component is shutdown and the zkClient is closed will the controller resign leadership.
There is a second problem in that a broker that does not successfully undergo controlled shutdown will also remain the leader for its partitions until the zkClient is shutdown, and the potential window there is large due to the aforementioned log manager shutdown.
It would be ideal if:
- controller leadership is resigned early in the shutdown process before request handling is stopped. Care will have to be taken so that the broker in question cannot regain it.
- we can reduce the window between an uncontrolled shutdown and resigning leadership of partitions through the zkclient close failsafe.