[HBASE-4633] Potential memory leak in client RPC timeout mechanism

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.90.3
• Component/s: Client

Environment

HBase version: 0.90.3 + Patches , Hadoop version: CDH3u0

• Assignee: Unassigned
• Reporter: Shrijeet Paliwal
• Created: 20/Oct/11 01:58
• Updated: 12/Jun/22 19:42
• Resolved: 16/Nov/16 21:02

Description

Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937,
https://issues.apache.org/jira/browse/HBASE-4003

We have been using the 'hbase.client.operation.timeout' knob
introduced in 2937 for quite some time now. It helps us enforce SLA.
We have two HBase clusters and two HBase client clusters. One of them
is much busier than the other.

We have seen a deterministic behavior of clients running in busy
cluster. Their (client's) memory footprint increases consistently
after they have been up for roughly 24 hours.
This memory footprint almost doubles from its usual value (usual case
== RPC timeout disabled). After much investigation nothing concrete
came out and we had to put a hack
which keep heap size in control even when RPC timeout is enabled. Also
note , the same behavior is not observed in 'not so busy
cluster.

The patch is here : https://gist.github.com/1288023

Issue Links

• is related to: HBASE-4956 Control direct memory buffer consumption by HBaseClient