Kafka
  1. Kafka
  2. KAFKA-1298

Controlled shutdown tool doesn't seem to work out of the box

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.2.0
    • Component/s: None
    • Labels:

      Description

      Download Kafka and try to use our shutdown tool. Got this:

      bin/kafka-run-class.sh kafka.admin.ShutdownBroker --zookeeper localhost:2181 --broker 0
      [2014-03-06 16:58:23,636] ERROR Operation failed due to controller failure (kafka.admin.ShutdownBroker$)
      java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: jkreps-mn.linkedin.biz; nested exception is:
      java.net.ConnectException: Connection refused]
      at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:340)
      at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:249)
      at kafka.admin.ShutdownBroker$.kafka$admin$ShutdownBroker$$invokeShutdown(ShutdownBroker.scala:56)
      at kafka.admin.ShutdownBroker$.main(ShutdownBroker.scala:109)
      at kafka.admin.ShutdownBroker.main(ShutdownBroker.scala)
      Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: jkreps-mn.linkedin.biz; nested exception is:
      java.net.ConnectException: Connection refused]
      at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
      at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
      at javax.naming.InitialContext.lookup(InitialContext.java:392)
      at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1888)
      at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1858)
      at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
      ... 4 more
      Caused by: java.rmi.ConnectException: Connection refused to host: jkreps-mn.linkedin.biz; nested exception is:
      java.net.ConnectException: Connection refused
      at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
      at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
      at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
      at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
      at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
      at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
      ... 9 more
      Caused by: java.net.ConnectException: Connection refused
      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:382)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:241)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:228)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:431)
      at java.net.Socket.connect(Socket.java:527)
      at java.net.Socket.connect(Socket.java:476)
      at java.net.Socket.<init>(Socket.java:373)
      at java.net.Socket.<init>(Socket.java:187)
      at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
      at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
      at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
      ... 14 more
      Oh god, RMI???!!!?

      Presumably this is because we stopped setting the JMX port by default. This is good because setting the JMX port breaks the quickstart which requires running multiple nodes on a single machine. The root cause imo is just using RMI here instead of our regular RPC.

      1. KAFKA-1298_2014-06-09_17:20:44.patch
        26 kB
        Sriharsha Chintalapani
      2. KAFKA-1298.patch
        166 kB
        Sriharsha Chintalapani
      3. KAFKA-1298.patch
        20 kB
        Sriharsha Chintalapani

        Issue Links

          Activity

          Hide
          Jun Rao added a comment -

          Thanks for the followup patch. Committed to trunk.

          Show
          Jun Rao added a comment - Thanks for the followup patch. Committed to trunk.
          Hide
          Sriharsha Chintalapani added a comment -

          @Jun Rao can you please check the latest patch. With this I am able to run the tests in
          Total time: 4 mins 45.842 secs
          Without the controlled shutdown patch the tests are taking around the same time.

          Show
          Sriharsha Chintalapani added a comment - @Jun Rao can you please check the latest patch. With this I am able to run the tests in Total time: 4 mins 45.842 secs Without the controlled shutdown patch the tests are taking around the same time.
          Hide
          Sriharsha Chintalapani added a comment -

          Updated reviewboard https://reviews.apache.org/r/22309/diff/
          against branch origin/trunk

          Show
          Sriharsha Chintalapani added a comment - Updated reviewboard https://reviews.apache.org/r/22309/diff/ against branch origin/trunk
          Hide
          Jun Rao added a comment -

          Sriharsha,

          That makes sense. Perhaps we could do the following.

          1. Make the change to skip StopReplica if replication factor is 1.
          2. Add an option in createBrokerConfig() to disable controlled shutdown.
          3. For all unit tests that create more than 1 replica, disable controlled shutdown when calling createBrokerConfig(). In unit tests, we need to stop all brokers during tearDown(). So, controlled shutdown is going to timeout.

          We can then see if the time to complete all unit tests is comparable to what we have before.

          Show
          Jun Rao added a comment - Sriharsha, That makes sense. Perhaps we could do the following. 1. Make the change to skip StopReplica if replication factor is 1. 2. Add an option in createBrokerConfig() to disable controlled shutdown. 3. For all unit tests that create more than 1 replica, disable controlled shutdown when calling createBrokerConfig(). In unit tests, we need to stop all brokers during tearDown(). So, controlled shutdown is going to timeout. We can then see if the time to complete all unit tests is comparable to what we have before.
          Hide
          Sriharsha Chintalapani added a comment -

          Jun Rao Neha Narkhede
          In case of AddPartitionsTest its running 4 brokers and creating 4 topics with a replication factor > 2. This is triggering leadership hand off when the test is stopping the servers which causing the latency and tearDown gets called for every test . This latency is there when i turned on controlled.shutdown without the patch.
          I tested skipping StopReplica when the replication factor is 1 didn't notice any issues with that. I can send patch for that.
          But to speed up the tests we need to disable controlled.shutdown.
          Thanks.

          Show
          Sriharsha Chintalapani added a comment - Jun Rao Neha Narkhede In case of AddPartitionsTest its running 4 brokers and creating 4 topics with a replication factor > 2. This is triggering leadership hand off when the test is stopping the servers which causing the latency and tearDown gets called for every test . This latency is there when i turned on controlled.shutdown without the patch. I tested skipping StopReplica when the replication factor is 1 didn't notice any issues with that. I can send patch for that. But to speed up the tests we need to disable controlled.shutdown. Thanks.
          Hide
          Sriharsha Chintalapani added a comment -

          Jun Rao Shouldn't we take the partition to offline state and stop replica. I will test this part again but when I was initially working on this I tested returning immediately if the replication factor is 1 that resulted in a error during the shutdown.
          Without this patch if I enable the controlled shutdown it does take the same time. Although in that case there is an exception happens and returns with controlledshutdown success.

          Show
          Sriharsha Chintalapani added a comment - Jun Rao Shouldn't we take the partition to offline state and stop replica. I will test this part again but when I was initially working on this I tested returning immediately if the replication factor is 1 that resulted in a error during the shutdown. Without this patch if I enable the controlled shutdown it does take the same time. Although in that case there is an exception happens and returns with controlledshutdown success.
          Hide
          Jun Rao added a comment -

          Sriharsha,

          Thanks for the patch. Do we know why the controlled shutdown take that long in the case with just one broker? I was thinking this should only add a few ms overhead. So, instead of turning controlled shutdown off in the unit tests, perhaps we should just improve the performance of controlled shutdown?

          Looking at the code, it seems if replication factor is 1, the controller can just send ack back immediately w/o having to send any requests like StopReplica.

          Show
          Jun Rao added a comment - Sriharsha, Thanks for the patch. Do we know why the controlled shutdown take that long in the case with just one broker? I was thinking this should only add a few ms overhead. So, instead of turning controlled shutdown off in the unit tests, perhaps we should just improve the performance of controlled shutdown? Looking at the code, it seems if replication factor is 1, the controller can just send ack back immediately w/o having to send any requests like StopReplica.
          Hide
          Sriharsha Chintalapani added a comment -

          Neha Narkhede Yes I added the config there. Thanks.
          Jun Rao Thanks for catching it.

          Show
          Sriharsha Chintalapani added a comment - Neha Narkhede Yes I added the config there. Thanks. Jun Rao Thanks for catching it.
          Hide
          Sriharsha Chintalapani added a comment -

          Created reviewboard https://reviews.apache.org/r/22309/diff/
          against branch origin/trunk

          Show
          Sriharsha Chintalapani added a comment - Created reviewboard https://reviews.apache.org/r/22309/diff/ against branch origin/trunk
          Hide
          Neha Narkhede added a comment -

          Sriharsha Chintalapani The unit tests use some utility in TestUtils to create the server config. In this utility, we can specifically turn off controlled shutdown. But let's leave it as on by default in KafkaConfig

          Show
          Neha Narkhede added a comment - Sriharsha Chintalapani The unit tests use some utility in TestUtils to create the server config. In this utility, we can specifically turn off controlled shutdown. But let's leave it as on by default in KafkaConfig
          Hide
          Sriharsha Chintalapani added a comment -

          Jun Rao This patch made controlled.shutdown.enable to true by default. So its kicking off controlled shutdown after each test in AddPartitonsTest which is causing the delay in execution. If I disable the controlled shutdown tests are passing in 22secs. I can send a patch by disabling controlled shutdown for AddPartitionsTest servers.

          Show
          Sriharsha Chintalapani added a comment - Jun Rao This patch made controlled.shutdown.enable to true by default. So its kicking off controlled shutdown after each test in AddPartitonsTest which is causing the delay in execution. If I disable the controlled shutdown tests are passing in 22secs. I can send a patch by disabling controlled shutdown for AddPartitionsTest servers.
          Hide
          Sriharsha Chintalapani added a comment -

          Jun Rao I am checking the test and will update with patch.

          Show
          Sriharsha Chintalapani added a comment - Jun Rao I am checking the test and will update with patch.
          Hide
          Jun Rao added a comment -

          Thanks for the patch. This patch seems to have made the unit tests significantly slower than before. For example, the following test took ~26 secs before the patch and more than 2 mins after the patch.

          ./gradlew -Dtest.single=AddPartitionsTest cleanTest core:test

          Show
          Jun Rao added a comment - Thanks for the patch. This patch seems to have made the unit tests significantly slower than before. For example, the following test took ~26 secs before the patch and more than 2 mins after the patch. ./gradlew -Dtest.single=AddPartitionsTest cleanTest core:test
          Hide
          Neha Narkhede added a comment -

          Thanks for the patches, pushed to trunk

          Show
          Neha Narkhede added a comment - Thanks for the patches, pushed to trunk
          Hide
          Sriharsha Chintalapani added a comment -

          Created reviewboard https://reviews.apache.org/r/21676/diff/
          against branch origin/trunk

          Show
          Sriharsha Chintalapani added a comment - Created reviewboard https://reviews.apache.org/r/21676/diff/ against branch origin/trunk
          Hide
          Sriharsha Chintalapani added a comment -

          Neha Narkhede Thanks for the details. I tested in few combinations by enabling controlled shutdown and controlled.shutdown.max.retries=100. In a single broker with replication factor set to 1 although there are exceptions thrown which are getting caught in PartitionStateMachine.handleStateChange and controlled shutdown is going through without blocking.
          1)KafkaApis.handleControlledShutdownRequest calls KafkaController.shutdownBroker which checks if the broker about to shutdown is the leader in the above case it is and invokes PartitionStateMachine.handleStateChange
          2) PartitionStateMachine.electLeaderForPartition calls ControlledShutdownLeaderSelector.selectLeader which checks if there are any other brokers available to make it leader in this case it will be empty and throws a StateChangeException which is being caught in handleStateChange logged into state-change.log
          3) KafkaController.shutdownBroker doesn't know about the exception goes forward with returning a successful controlledShutdown response back.
          4) KafkaController checks for the remainingPartitions
          leaderIsrAndControllerEpoch.leaderAndIsr.leader == id && controllerContext.partitionReplicaAssignment(topicAndPartition).size > 1.

          I tested in multi node env and its the same case.
          so by making controlled shutdown default it won't be blocking at this point.
          But I'll send a patch to skip leadership handoff if the replicationFactor is 1

          Show
          Sriharsha Chintalapani added a comment - Neha Narkhede Thanks for the details. I tested in few combinations by enabling controlled shutdown and controlled.shutdown.max.retries=100. In a single broker with replication factor set to 1 although there are exceptions thrown which are getting caught in PartitionStateMachine.handleStateChange and controlled shutdown is going through without blocking. 1)KafkaApis.handleControlledShutdownRequest calls KafkaController.shutdownBroker which checks if the broker about to shutdown is the leader in the above case it is and invokes PartitionStateMachine.handleStateChange 2) PartitionStateMachine.electLeaderForPartition calls ControlledShutdownLeaderSelector.selectLeader which checks if there are any other brokers available to make it leader in this case it will be empty and throws a StateChangeException which is being caught in handleStateChange logged into state-change.log 3) KafkaController.shutdownBroker doesn't know about the exception goes forward with returning a successful controlledShutdown response back. 4) KafkaController checks for the remainingPartitions leaderIsrAndControllerEpoch.leaderAndIsr.leader == id && controllerContext.partitionReplicaAssignment(topicAndPartition).size > 1. I tested in multi node env and its the same case. so by making controlled shutdown default it won't be blocking at this point. But I'll send a patch to skip leadership handoff if the replicationFactor is 1
          Hide
          Neha Narkhede added a comment -

          @sriharsha I'm not sure that we've fixed the single replica problem with controlled shutdown. I think what we need to do for this JIRA is-
          1. Fix the single replica controlled shutdown issue
          2. Remove the jmx hook from KafkaController
          3. Remove the ShutdownBroker tool
          4. Default to controlled shutdown in the broker config.

          Show
          Neha Narkhede added a comment - @sriharsha I'm not sure that we've fixed the single replica problem with controlled shutdown. I think what we need to do for this JIRA is- 1. Fix the single replica controlled shutdown issue 2. Remove the jmx hook from KafkaController 3. Remove the ShutdownBroker tool 4. Default to controlled shutdown in the broker config.
          Hide
          Sriharsha Chintalapani added a comment -

          I am looking to work on this bug . Based on the comments above the consensus seems to be making controlled shutdown enable by default and making controlled shutdown of the single broker possible. I tested this on code from trunk I ran single broker with controlled shutdown enabled and sent kill -15 broker_pid I do see
          [Kafka Server 0], shutting down (kafka.server.KafkaServer)
          [Kafka Server 0], Starting controlled shutdown (kafka.server.KafkaServer)
          [Kafka Server 0], Controlled shutdown succeeded (kafka.server.KafkaServer)
          Do we still want to fix ShutdownBorker?.

          Show
          Sriharsha Chintalapani added a comment - I am looking to work on this bug . Based on the comments above the consensus seems to be making controlled shutdown enable by default and making controlled shutdown of the single broker possible. I tested this on code from trunk I ran single broker with controlled shutdown enabled and sent kill -15 broker_pid I do see [Kafka Server 0] , shutting down (kafka.server.KafkaServer) [Kafka Server 0] , Starting controlled shutdown (kafka.server.KafkaServer) [Kafka Server 0] , Controlled shutdown succeeded (kafka.server.KafkaServer) Do we still want to fix ShutdownBorker?.
          Hide
          Joel Koshy added a comment -

          After thinking a bit more - the command-line tool in its current form even with the fix is kind of useless. i.e., it only moves leadership and does not kill the process. So the permissions thing doesn't really apply.

          Show
          Joel Koshy added a comment - After thinking a bit more - the command-line tool in its current form even with the fix is kind of useless. i.e., it only moves leadership and does not kill the process. So the permissions thing doesn't really apply.
          Hide
          Jay Kreps added a comment -

          Cool so for 0.8.1 I just want to document what you need to do to make the tool work (if it can ever work) or else just not mention it and just document the config.

          Post 0.8.1 we can do both. I agree that if we fixed the config then we can default it on and most people won't need to know about it.

          If we also fix the tool then I agree having a remote shutdown command-line tool is kind of nice, though, WRT permissions, I suppose if you don't have permission to kill something there may be a reason for that or else you would change the permission? Another way to say that is it is a little sketchy that there is an API with no permissions to kill the broker...

          Show
          Jay Kreps added a comment - Cool so for 0.8.1 I just want to document what you need to do to make the tool work (if it can ever work) or else just not mention it and just document the config. Post 0.8.1 we can do both. I agree that if we fixed the config then we can default it on and most people won't need to know about it. If we also fix the tool then I agree having a remote shutdown command-line tool is kind of nice, though, WRT permissions, I suppose if you don't have permission to kill something there may be a reason for that or else you would change the permission? Another way to say that is it is a little sketchy that there is an API with no permissions to kill the broker...
          Hide
          Neha Narkhede added a comment -

          One fix would be to change the definition of controlled shutdown such that we only blocks on leadership handoff if there are other replicas that we CAN hand off to (otherwise what's the point of blocking...another replica isn't likely to magically appear).

          +1.

          I agree that this is not something that we can fix in 0.8.1

          Show
          Neha Narkhede added a comment - One fix would be to change the definition of controlled shutdown such that we only blocks on leadership handoff if there are other replicas that we CAN hand off to (otherwise what's the point of blocking...another replica isn't likely to magically appear). +1. I agree that this is not something that we can fix in 0.8.1
          Hide
          Joel Koshy added a comment -

          I think the tool itself may be useful to have (say if you don't have the permissions to issue a SIGTERM to a broker PID).

          To fix this, we can just have the tool send the ControlledShutdownRequest to the controller and read back the response and decide whether to retry.

          Show
          Joel Koshy added a comment - I think the tool itself may be useful to have (say if you don't have the permissions to issue a SIGTERM to a broker PID). To fix this, we can just have the tool send the ControlledShutdownRequest to the controller and read back the response and decide whether to retry.
          Hide
          Sriram Subramanian added a comment -

          I dont think we have enabled it by default which we should. The issue is today the controller does not let a broker to shutdown if it is the only one up. We try to move the leader for all the topic partitions on a broker and we fail if there are no other brokers to move the leadership to. We should probably succeed if the number of replicas = 1 because there is not much the controller can do. This however may not be something we can fix in 0.8.1.

          Show
          Sriram Subramanian added a comment - I dont think we have enabled it by default which we should. The issue is today the controller does not let a broker to shutdown if it is the only one up. We try to move the leader for all the topic partitions on a broker and we fail if there are no other brokers to move the leadership to. We should probably succeed if the number of replicas = 1 because there is not much the controller can do. This however may not be something we can fix in 0.8.1.
          Hide
          Jay Kreps added a comment -

          In the short term I need to document in because that configuration defaults off in 0.8.1.

          But yeah I think this is a good question. One point Jun raised was that enabling this by default is a little problematic because the common test scenario is using a single-node cluster with replication-factor 1. In this scenario you actually can't shut down at all. One fix would be to change the definition of controlled shutdown such that we only blocks on leadership handoff if there are other replicas that we CAN hand off to (otherwise what's the point of blocking...another replica isn't likely to magically appear).

          Show
          Jay Kreps added a comment - In the short term I need to document in because that configuration defaults off in 0.8.1. But yeah I think this is a good question. One point Jun raised was that enabling this by default is a little problematic because the common test scenario is using a single-node cluster with replication-factor 1. In this scenario you actually can't shut down at all. One fix would be to change the definition of controlled shutdown such that we only blocks on leadership handoff if there are other replicas that we CAN hand off to (otherwise what's the point of blocking...another replica isn't likely to magically appear).
          Hide
          Neha Narkhede added a comment -

          Do we even need this tool anymore? Currently controlled shutdown is enabled through a config and automatically kicks in during a kill -15 of a Kafka server. I almost think we can get rid of a tool to do controlled shutdown.

          Show
          Neha Narkhede added a comment - Do we even need this tool anymore? Currently controlled shutdown is enabled through a config and automatically kicks in during a kill -15 of a Kafka server. I almost think we can get rid of a tool to do controlled shutdown.

            People

            • Assignee:
              Sriharsha Chintalapani
              Reporter:
              Jay Kreps
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development