Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-5550

[memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.1.0-SNAPSHOT
    • None
    • Core/Engine
    • None
    • 2023-2-Storage, 2023-3-Storage

    Description

      [memory]OOM occurs when insert less than 1M data with benchmark and default configurations of IoTDB

      build:
      master 1a69b40 1.1.0-SNAPSHOT

      reproduction:
      1. start 1C1D with the configurations of attachment
      2. run benchmarks to insert data into IoTDB

      datanode log:

      2023-02-17 10:56:32,311 [pool-24-IoTDB-DataNodeInternalRPC-Processor-1] ERROR o.a.i.c.c.t.WrappedThreadPoolExecutor:108 - Exception in thread pool org.apache.iotdb.threadpool:type=DataNodeInternalRPC-Processor
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      2023-02-17 10:56:21,422 [pool-19-IoTDB-ratis-bg-disk-guardian-1] ERROR o.a.i.c.c.t.ScheduledExecutorUtil:191 - Run thread failed
      java.lang.OutOfMemoryError: GC overhead limit exceeded
      2023-02-17 10:56:50,734 [pool-23-IoTDB-MPPCoordinatorScheduled-9] WARN  o.a.i.d.m.p.s.SimpleQueryTerminator:108 - cancel query 20230217_025125_00644_1 on node TEndPoint(ip:iotdb-44, port:10730) failed.
      org.apache.thrift.TException: Error in calling method cancelQuery, because: GC overhead limit exceeded
              at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:57)
              at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$c055cf81.cancelQuery(<generated>)
              at org.apache.iotdb.db.mpp.plan.scheduler.SimpleQueryTerminator.syncTerminate(SimpleQueryTerminator.java:102)
              at org.apache.iotdb.commons.concurrent.WrappedCallable$1.callMayThrow(WrappedCallable.java:46)
              at org.apache.iotdb.commons.concurrent.WrappedCallable.call(WrappedCallable.java:31)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
      

      disk taken:

      4.1M	./datanode/consensus
      28K	./datanode/data
      40M	./datanode/wal
      64K	./datanode/system
      45M	./datanode
      4.1M	./confignode/consensus
      24K	./confignode/system
      4.1M	./confignode
      49M	.
      

      metric monitor:

      Attachments

        1. IOTDB-5550.tar.gz
          121 kB
          changxue
        2. image-2023-02-17-14-28-57-874.png
          159 kB
          changxue

        Activity

          People

            xingtanzjr Jinrui Zhang
            changxue changxue
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: