[HBASE-6290] Add a function a mark a server as dead and start the recovery the process

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.95.2
• Fix Version/s: 2.0.0
• Component/s: monitoring
• Labels: beginner
• Hadoop Flags: Reviewed

Release Note

Adds a script to mark a server as dead.

Usage: considerAsDead.sh --hostname serverName

• Tags: beginner
• Assignee: Talat Uyarer
• Reporter: Nicolas Liochon
• Created: 28/Jun/12 16:39
• Updated: 13/Jun/22 16:55
• Resolved: 29/Sep/14 17:23

Description

ZooKeeper is used a a monitoring tool: we use znode and we start the recovery process when a znode is deleted by ZK because it got a timeout. This timeout is defaulted to 90 seconds, and often set to 30s

However, some HW issues could be detected by specialized hw monitoring tools before the ZK timeout. For this reason, it makes sense to offer a very simple function to mark a RS as dead. This should not take in

It could be a hbase shell function such as
considerAsDead ipAddress|serverName

This would delete all the znodes of the server running on this box, starting the recovery process.

Such a function would be easily callable (at callers risk) by any fault detection tool... We could have issues to identify the right master & region servers around ipv4 vs ipv6 vs and multi networked boxes however.

Issue Links

• is required by: HBASE-5843 Improve HBase MTTR - Mean Time To Recover