[AMBARI-21068] Kafka broker goes down after Ambari upgrade from 2.5.0 to 2.5.1 due to missing 'kafka.timeline.metrics.instanceId' property

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.5.1
• Fix Version/s: 2.5.1
• Component/s: ambari-upgrade
• Labels: upgrade
• Assignee: Dmytro Sen
• Reporter: Vivek Sharma
• Created: 19/May/17 07:44
• Updated: 20/May/17 00:55
• Resolved: 19/May/17 23:41

Description

STR:
1. Deploy HDP-2.6.0 cluster with Ambari-2.5.0.3
2. Upgrade Ambari to 2.5.1.0-130 (hash: e7745e411638405dd0a406305e1da222c6879daa)
3. Observed that Kafka broker went down

Kafka logs indicate:

[2017-05-19 03:42:17,105] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: Missing required property 'kafka.timeline.metrics.instanceId'
        at scala.Predef$.require(Predef.scala:233)
        at kafka.utils.VerifiableProperties.getString(VerifiableProperties.scala:177)
        at org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter.init(KafkaTimelineMetricsReporter.java:169)
        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:66)
        at kafka.metrics.KafkaMetricsReporter$$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:64)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
        at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:64)
        at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:27)
        at kafka.Kafka$.main(Kafka.scala:58)
        at kafka.Kafka.main(Kafka.scala)

Did not see this property being added on ambari_upgrade_config_changes.json file

Issue Links

• is duplicated by: AMBARI-21075 Kafka broker goes down after Ambari upgrade from 2.5.0 to 2.5.1 due to missing 'kafka.timeline.metrics.instanceId' property
• links to: Review Board
• links to: Review Board 2