[HBASE-13714] Add tracking of the total response queue size

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.0.2, 1.2.0, 2.0.0
• Component/s: IPC/RPC, master, metrics, regionserver
• Assignee: Unassigned
• Reporter: Esteban Gutierrez
• Created: 19/May/15 20:25
• Updated: 24/Jun/22 18:59
• Resolved: 11/Jun/22 20:03

Description

I noticed this behavior while working on HBASE-13694:
Once we are done processing a request, we decrement the call queue size on the RPC server. However, responses can be very large and sometimes sending them can take a long time. Since we don't keep track the response queue via metrics it is hard to spot when the responses are using too much resources on the RS.

Ideally we should be tracking on the RS how much data we have in-flight in the response queue via metrics and not just in the logs if the size of the response exceeds a threshold (e.g hbase.ipc.warn.response.size or hbase.ipc.warn.response.time)