[AMBARI-12678] Installing additional Kafka broker on node with hostname that comes alphabetically before existing broker fails to start

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.7.0, 2.0.0, 2.1.0
• Fix Version/s: 2.2.0
• Component/s: ambari-server
• Assignee: Harsha
• Reporter: Carita Ou
• Created: 07/Aug/15 17:41
• Updated: 11/Aug/16 13:28
• Resolved: 18/Nov/15 19:44

Description

1. Install Ambari with 3 nodes, example hostname01, hostname02, hostname03
2. Install Kafka service on hostname03. Kafka starts successfully
3. Go to hostname02 and add Kafka Broker. Start kafka broker and it will fail with below error:

[2015-08-05 17:13:51,306] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
		at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:174)
		at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:58)
		at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
		at kafka.server.KafkaServer.startup(KafkaServer.scala:124)
		at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
		at kafka.Kafka$.main(Kafka.scala:46)
		at kafka.Kafka.main(Kafka.scala)

The 2nd broker added on hostname02 failed to start because it was assigned the same brokerid as the broker installed on hostname03. The brokerid is assigned to a broker in kafka.py:

brokerid = str(sorted(params.kafka_hosts).index(params.hostname))
kafka_server_config['broker.id'] = brokerid

params.py gets the kafka_hosts from clusterHostInfo:

kafka_hosts = config['clusterHostInfo']['kafka_broker_hosts']

clusterHostInfo maintains the list of kafka brokers in order:

"kafka_broker_hosts": [
	"hostname02",
	"hostname03"
],

Since the brokerid is the index of the hostname, and the broker on hostname03 was added first, it was at index 0. After adding a new broker on hostname02, it was inserted to the list at index 0 and hostname03 was moved to index 1. Because the broker on hostname03 was not restarted, it was still running with brokerid 0, so the broker on hostname02 with brokerid 0 was unable to start.

Workaround: Restart all brokers