Major Description
In KafkaServer.controlledShutdown(), it initiates a controlled shutdown, and then if there's a failure, it will retry the controlledShutdown.
Looking at the code, there are 2 ways a retry could fail, one with an error response from the controller, and this messaging code:
info("Remaining partitions to move: %s".format(shutdownResponse.partitionsRemaining.mkString(","))) info("Error code from controller: %d".format(shutdownResponse.errorCode))
Alternatively, there could be an IOException, with this code executed:
catch { case ioe: java.io.IOException => channel.disconnect() channel = null // ignore and try again }
And then finally, in either case:
if (!shutdownSuceeded) { Thread.sleep(config.controlledShutdownRetryBackoffMs) warn("Retrying controlled shutdown after the previous attempt failed...") }
It would be nice if the nature of the IOException were logged in either case (I'd be happy with an ioe.getMessage() instead of a full stack trace, as kafka in general tends to be too willing to dump IOException stack traces!).
I suspect, in my case, the actual IOException is a socket timeout (as the time between initial "Starting controlled shutdown...." and the first "Retrying..." message is usually about 35 seconds (the socket timeout + the controlled shutdown retry backoff). So, it would seem that really, the issue in this case is that controlled shutdown is taking too long. It would seem sensible instead to have the controller report back to the server (before the socket timeout) that more time is needed, etc.