[HBASE-16165] Decrease RpcServer.callQueueSize before writeResponse causes OOM

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.0, 1.1.6, 1.2.3, 0.98.22, 2.0.0
• Fix Version/s: 1.3.0, 1.1.7, 0.98.23, 1.2.4, 2.0.0
• Component/s: IPC/RPC, rpc
• Labels: trivial
• Hadoop Flags: Reviewed
• Assignee: Guanghao Zhang
• Reporter: Duo Zhang
• Created: 01/Jul/16 09:53
• Updated: 09/Nov/17 01:35
• Resolved: 18/Sep/16 02:34

Description

In RpcServer, we use callQueueSizeInBytes to avoid queuing too many calls which causes OOM. But in CallRunner.run, we decrease it before send the response back. And even after calling sendResponseIfReady, the call object could stay in our heap for a long time if we can not write out the response(That's why we need a Responder thread...). This makes it possible that the actual size of all call object in heap is larger than maxQueueSizeInBytes and causes OOM.