[KAFKA-3091] Broker with an invalid id would not start when its id is updated to a new valid one

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.9.0.0
• Fix Version/s: 0.10.0.0
• Assignee: Grant Henke
• Reporter: Vahid Hashemian
• Created: 12/Jan/16 22:13
• Updated: 05/Feb/16 21:06
• Resolved: 19/Jan/16 02:56

Description

Setup being used:

• Kafka 0.9.0
• Two servers:
  • server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
  • server 2: 2 brokers (ids: 10 and 1 -> 11)

When on server 2 the second broker with initial id of 1 is started an error returned indicating the id 1 is in use. When the corresponding servers.properties file is updated and id is changed to 11 the broker would no longer start. The following error is returned:

[2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
[2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed (org.apache.zookeeper.ZooKeeper)
[2016-01-12 13:40:22,155] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
[2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)

I looked at existing JIRA tickets referencing this error but none seemed to describe the exact scenario as the one here. Restarting brokers or the zookeeper would not seem to help.

And upon further testing it appears the number of servers is irrelevant here, and the same issue would surface in a one server setup too.

Issue Links

• relates to: KAFKA-3012 Avoid reserved.broker.max.id collisions on upgrade