Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1108

when controlled shutdown attempt fails, the reason is not always logged

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.2.0
    • Component/s: None
    • Labels:

      Description

      In KafkaServer.controlledShutdown(), it initiates a controlled shutdown, and then if there's a failure, it will retry the controlledShutdown.

      Looking at the code, there are 2 ways a retry could fail, one with an error response from the controller, and this messaging code:

      info("Remaining partitions to move: %s".format(shutdownResponse.partitionsRemaining.mkString(",")))
      info("Error code from controller: %d".format(shutdownResponse.errorCode))
      

      Alternatively, there could be an IOException, with this code executed:

                  catch {
                    case ioe: java.io.IOException =>
                      channel.disconnect()
                      channel = null
                      // ignore and try again
                  }
      

      And then finally, in either case:

                if (!shutdownSuceeded) {
                  Thread.sleep(config.controlledShutdownRetryBackoffMs)
                  warn("Retrying controlled shutdown after the previous attempt failed...")
                }
      

      It would be nice if the nature of the IOException were logged in either case (I'd be happy with an ioe.getMessage() instead of a full stack trace, as kafka in general tends to be too willing to dump IOException stack traces!).

      I suspect, in my case, the actual IOException is a socket timeout (as the time between initial "Starting controlled shutdown...." and the first "Retrying..." message is usually about 35 seconds (the socket timeout + the controlled shutdown retry backoff). So, it would seem that really, the issue in this case is that controlled shutdown is taking too long. It would seem sensible instead to have the controller report back to the server (before the socket timeout) that more time is needed, etc.

        Attachments

        1. KAFKA-1108.patch
          1.0 kB
          Ewen Cheslack-Postava
        2. KAFKA-1108_2014-10-16_13:53:11.patch
          3 kB
          Ewen Cheslack-Postava

          Activity

            People

            • Assignee:
              ewencp Ewen Cheslack-Postava
              Reporter:
              jbrosenberg@gmail.com Jason Rosenberg
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: