[HBASE-18549] Unclaimed replication queues can go undetected

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 1.4.8, 2.1.1
• Component/s: Replication
• Hadoop Flags: Reviewed
• Assignee: Xu Cang
• Reporter: Ashu Pachauri
• Created: 09/Aug/17 20:58
• Updated: 01/Feb/19 19:55
• Resolved: 02/Oct/18 01:51

Description

We have come across this situation multiple times where a zookeeper issues can cause NodeFailoverWorker to fail picking up replication queue for a dead region server silently. One example is when the znode size for a particular queue exceed jute.maxBuffer value.

There can be other situations that may lead to this and just go undetected. We need to have a metric for number of unclaimed replication queues. This will help in mitigating the problem through alerting on the metric and identifying underlying issues.