[AMBARI-24203] Improve Kafka service check to check for under-replicated partitions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.7.0
• Component/s: stacks
• Labels: pull-request-available
• Assignee: Yolanda M. Davis
• Reporter: Yolanda M. Davis
• Created: 28/Jun/18 01:40
• Updated: 01/Oct/18 14:48
• Resolved: 02/Jul/18 18:55

Description

During Ambari Rolling Upgrade, we invoke Kafka Service check to ensure that Kafka is healthy. The current implementation only does basic topic creation/deletion. We need to extend the implementation to check for number of under-replicated partitions.

We have this support in kafka-topics.sh script which is already used in the kafka service check script. We need to extend above script to call below command

sh kafka-topics.sh --describe --zookeeper localhost:2181 --under-replicated-partitions

If the command output is empty, then there are no under replicated partotions.
If the output contains "Topic:" string, then there are some under replicated partitions, we can fail the service check.

sh kafka-topics.sh --describe --zookeeper localhost:2181 --under-replicated-partitions
 Topic: TEST Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1
 Topic: TEST Partition: 1 Leader: 1 Replicas: 2,1 Isr: 1

Issue Links

• is related to: AMBARI-24719 Kafka Rolling Restart causes outage(s) due to not checking for under replicated partitions
• links to: GitHub Pull Request #1635