[IOTDB-5550] [memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.1.0-SNAPSHOT
• Component/s: Core/Engine
• Sprint: 2023-2-Storage, 2023-3-Storage
• Assignee: Jinrui Zhang
• Reporter: changxue
• Created: 17/Feb/23 06:30
• Updated: 06/Mar/23 07:39

Description

[memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB

build:
master 1a69b40 1.1.0-SNAPSHOT

reproduction:
1. start 1C1D with the configurations of attachment
2. run benchmarks to insert data into IoTDB

datanode log:

2023-02-17 10:56:32,311 [pool-24-IoTDB-DataNodeInternalRPC-Processor-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-02-17 10:56:21,422 [pool-19-IoTDB-ratis-bg-disk-guardian-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:191 - Run thread failed
java.lang.OutOfMemoryError: GC overhead limit exceeded
2023-02-17 10:56:50,734 [pool-23-IoTDB-MPPCoordinatorScheduled-9] WARN  o.a.i.d.m.p.s.SimpleQueryTerminator:108 - cancel query 20230217_025125_00644_1 on node TEndPoint(ip:iotdb-44, port:10730) failed.
org.apache.thrift.TException: Error in calling method cancelQuery, because: GC overhead limit exceeded
        at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:57)
        at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$c055cf81.cancelQuery(<generated>)
        at org.apache.iotdb.db.mpp.plan.scheduler.SimpleQueryTerminator.syncTerminate(SimpleQueryTerminator.java:102)
        at org.apache.iotdb.commons.concurrent.WrappedCallable$1.callMayThrow(WrappedCallable.java:46)
        at org.apache.iotdb.commons.concurrent.WrappedCallable.call(WrappedCallable.java:31)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

disk taken:

4.1M	./datanode/consensus
28K	./datanode/data
40M	./datanode/wal
64K	./datanode/system
45M	./datanode
4.1M	./confignode/consensus
24K	./confignode/system
4.1M	./confignode
49M	.

metric monitor: