Description
When a zookeeper ensemble has 'observers', the zkstatus page incorrectly says Zookeeper status is in error (See attachment.)
This is because the ZookeeperStatusHandler does not account for the 'observer' role whatsoever.
This should be an easy fix - I see there being two options;
1. Treat observers as followers by changing L112 to
if ("follower".equals(state) || "observer".equals(state)) {
2. Ignore observers from the required follower count by changing L116 to
reportedFollowers = Integer.parseInt(String.valueOf(stat.get("zk_synced_followers")));
Option 1 will make the zkstatus page show error when an observer is down.
Option 2 will not make the zkstatus page show error when an observer is down.
Ideally, additional logic to account for observers should be added, and show a STATUS_YELLOW when any observers are down (but followers are all up), as this means the ensemble is only in a degraded, but functional state.
Happy to create a PR, however I don't have a lot of free time at home at the moment, so it may take a week or two.
Additional info:
See below for example mntr output for the Leader/Follower/Observer roles, noting the Leader's zk_followers and zk_synced_followers values, and the values of zk_server_state.
Leader:
[root@master1 ~]# echo mntr | nc master3 12181 zk_version 3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMT zk_avg_latency 0 zk_max_latency 2 zk_min_latency 0 zk_packets_received 97 zk_packets_sent 96 zk_num_alive_connections 2 zk_outstanding_requests 0 zk_server_state leader zk_znode_count 92 zk_watch_count 7 zk_ephemerals_count 9 zk_approximate_data_size 236333 zk_open_file_descriptor_count 64 zk_max_file_descriptor_count 4096 zk_followers 4 zk_synced_followers 2 zk_pending_syncs 0 zk_last_proposal_size -1 zk_max_proposal_size -1 zk_min_proposal_size -1
Follower:
[root@master1 ~]# echo mntr | nc master2 12181 zk_version 3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMT zk_avg_latency 0 zk_max_latency 6 zk_min_latency 0 zk_packets_received 97 zk_packets_sent 96 zk_num_alive_connections 2 zk_outstanding_requests 0 zk_server_state follower zk_znode_count 92 zk_watch_count 7 zk_ephemerals_count 9 zk_approximate_data_size 236333 zk_open_file_descriptor_count 60 zk_max_file_descriptor_count 4096
Observer:
[root@master1 ~]# echo mntr | nc slave1 12181 zk_version 3.5.6-c11b7e26bc554b8523dc929761dd28808913f091, built on 10/08/2019 20:18 GMT zk_avg_latency 0 zk_max_latency 8 zk_min_latency 0 zk_packets_received 174 zk_packets_sent 173 zk_num_alive_connections 2 zk_outstanding_requests 0 zk_server_state observer zk_znode_count 92 zk_watch_count 7 zk_ephemerals_count 9 zk_approximate_data_size 236333 zk_open_file_descriptor_count 59 zk_max_file_descriptor_count 4096