Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Cannot Reproduce
-
0.98.10.1
-
None
-
None
-
None
Description
I just came across in interesting exception:
Caused by: java.io.IOException: Call 10 not added as the connection newbunny/127.0.0.1:60020/ClientService/lars (auth:SIMPLE)/60000 is closing
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.addCall(RpcClient.java:495)
at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1534)
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
... 13 more
Called from here:
at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:291)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:160)
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:115)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:91)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:247)
This happened when I scanned with multiple client against a single region server when all data is filtered at the server by a filter.
I had 10 clients, the region server has 30 handles.
This means the scanners are not getting closed and their lease has to expire.
The workaround is to increase hbase.ipc.client.connection.maxidletime.
But it's strange that this only happens at close time. And since I am not using up all handlers there shouldn't be any starvation.