Critical Description
Just ran into this on the Phoenix side.
We retrieve a Connection via RegionCoprocessorEnvironment.createConnection... getTable(...). And then call get on that table. The Get's key happens to be local. Now each call to table.get() leaves an open StoreScanner around forever. (verified with a memory profiler).
There references are held via RegionScannerImpl.storeHeap.scannersForDelayedClose. Eventially the RegionServer goes into a GC of death and can only ended with kill -9.
The reason appears to be that in this case there is no currentCall context. Some time in 2.x the Rpc handler/call was made responsible for closing open region scanners, but we forgot to handle ShortCircuitingClusterConnection
It's not immediately clear how to fix this. But it does make ShortCircuitingClusterConnection useless and dangerous. If you use it, you will create a giant memory leak.
Attachments
Issue Links
- relates to
-
HBASE-26869 RSRpcServices.scan should deep clone cells when RpcCallContext is null
-
- Resolved
-
-
-
- Resolved
-
- links to