[HBASE-21147] (1.4) Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.0.0, 3.0.0-alpha-1, 2.0.0
• Fix Version/s: 1.4.8
• Component/s: canary, Zookeeper
• Hadoop Flags: Reviewed
• Assignee: David Manning
• Reporter: David Manning
• Created: 04/Sep/18 02:50
• Updated: 05/Sep/18 00:50
• Resolved: 05/Sep/18 00:50

Description

When running org.apache.hadoop.hbase.tool.Canary with args -zookeeper -treatFailureAsError, the Canary will try to get a znode from each ZooKeeper server in the ensemble. If any server is unavailable or unresponsive, the canary will exit with a failure code.

If we use the Canary to gauge server health, and alert accordingly, this can be too strict. For example, in a 5-node ZooKeeper cluster, having one node down is safe and expected in rolling upgrades/patches.

This is a request to allow the Canary to take another parameter

-permittedZookeeperFailures <N>

If N=1, in the 5-node ZooKeeper ensemble example, then the Canary will still pass if 4 ZooKeeper nodes are reachable, but fail if 3 or fewer are reachable.

(This is my first Jira posting... sorry if I messed anything up.)

Issue Links

• is a clone of: HBASE-21126 Add ability for HBase Canary to ignore a configurable number of ZooKeeper down nodes