[HBASE-15155] Show All RPC handler tasks stops working after cluster is under heavy load for a while

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.98.0, 1.0.0, 0.94.19
• Component/s: monitoring
• Assignee: Tianying Chang
• Reporter: Tianying Chang
• Created: 21/Jan/16 23:04
• Updated: 17/Mar/16 20:00

Description

After we upgrade from 94.7 to 94.26 and 1.0, we found that "Show All RPC handler status" link on RS webUI stops working after running in production cluster with relatively high load for several days.

Turn out to be it is a bug introduced by https://issues.apache.org/jira/browse/HBASE-10312 The BoundedFIFOBuffer cause RPCHandler Status overriden/removed permanently when there is a spike of non-RPC tasks status that is over the MAX_SIZE (1000). So as long as the RS experienced "high" load once, the RPC status monitoring is gone forever, until RS is restarted.

We added a unit test that can repro this. And the fix can pass the test.