[HBASE-27941] Possible memory leak in MemStoreLAB implementation

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: in-memory-compaction, regionserver
• Assignee: Unassigned
• Reporter: Duo Zhang
• Created: 19/Jun/23 15:35
• Updated: 24/Feb/24 15:43

Description

We got this error message when running ITBLL against branch-3.

2023-06-09 14:44:15,386 ERROR [regionserver/core-1-2:16020-shortCompactions-0] util.ResourceLeakDetector: LEAK: RefCnt.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
        org.apache.hadoop.hbase.nio.RefCnt.<init>(RefCnt.java:59)
        org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
        org.apache.hadoop.hbase.regionserver.MemStoreLABImpl.<init>(MemStoreLABImpl.java:108)
        sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:55)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:43)
        org.apache.hadoop.hbase.regionserver.MemStoreLAB.newInstance(MemStoreLAB.java:116)
        org.apache.hadoop.hbase.regionserver.SegmentFactory.createMutableSegment(SegmentFactory.java:81)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.resetActive(AbstractMemStore.java:93)
        org.apache.hadoop.hbase.regionserver.AbstractMemStore.<init>(AbstractMemStore.java:83)
        org.apache.hadoop.hbase.regionserver.DefaultMemStore.<init>(DefaultMemStore.java:79)
        sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:55)
        org.apache.hadoop.hbase.util.ReflectionUtils.newInstance(ReflectionUtils.java:92)
        org.apache.hadoop.hbase.regionserver.HStore.getMemstore(HStore.java:377)
        org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:283)
        org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:6904)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1173)
        org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1170)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        java.util.concurrent.FutureTask.run(FutureTask.java:266)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        java.lang.Thread.run(Thread.java:750)

Need to dig more.

Issue Links

• is blocked by: HBASE-28398 Make sure we close all the scanners in TestHRegion
• relates to: HBASE-27919 Run ITBLL for branch-3
• relates to: HBASE-28285 Resource leak for memstore and flush

Sub-Tasks

• Introduce a close method for memstore for release active segment: Resolved
• Revisit the way we do reference counting on MemStoreLAB: Open