[IOTDB-3262] [ cluster metadata ] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 201 is too large...

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0-SNAPSHOT
• Component/s: Core/Schema Manager, mpp-cluster
• Labels: pull-request-available
• Sprint: 2022-6-Cluster
• Assignee: Yukun Zhou
• Reporter: 刘珍
• Created: 23/May/22 06:47
• Updated: 04/Aug/22 03:02
• Resolved: 25/Jun/22 01:20

Description

master_0523_51e9703
3C3D
创建30sg，300dev，1k ts/dev，共30万序列。
datanode内存：-Xmx7968M
创建元数据失败：
confignode 没有error，有WARN：
2022-05-23 13:53:31,618 [192.168.130.3-22278@group-000000000000->192.168.130.5-22278-GrpcLogAppender-LogAppenderDaemon] WARN o.a.r.g.s.GrpcLogAppender:177 - 192.168.130.3-22278@group-000000000000->192.168.130.5-22278-GrpcLogAppender: Wait interrupted by java.lang.InterruptedException
2022-05-23 13:58:21,715 [192.168.130.3-22278@group-000000000000-StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 201 is too large...
2022-05-23 13:58:21,718 [192.168.130.3-22278@group-000000000000-StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 432 is too large...
2022-05-23 13:58:21,743 [192.168.130.3-22278@group-000000000000-StateMachineUpdater] WARN o.a.i.d.m.r.MemoryStatistics:80 - Current series number 663 is too large...

datanode 报错：
2022-05-23 13:58:25,843 [pool-18-IoTDB-MPPCoordinatorWrite-5] ERROR o.a.i.c.c.s.SyncThriftClientWithErrorHandler:80 - root cause message Internal error processing sendFragmentInstance, LocalizedMessage Internal error processing sendFragmentInstance,
org.apache.thrift.TApplicationException: Internal error processing sendFragmentInstance
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$receiveBase$51(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.receiveBase(<generated>)
at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.recv_sendFragmentInstance(InternalService.java:170)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$recv_sendFragmentInstance$12(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.recv_sendFragmentInstance(<generated>)
at org.apache.iotdb.mpp.rpc.thrift.InternalService$Client.sendFragmentInstance(InternalService.java:157)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.CGLIB$sendFragmentInstance$39(<generated>)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446$$FastClassByCGLIB$$76030a70.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.intercept(SyncThriftClientWithErrorHandler.java:55)
at org.apache.iotdb.commons.client.sync.SyncDataNodeInternalServiceClient$$EnhancerByCGLIB$$fa724446.sendFragmentInstance(<generated>)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchRemote(FragmentInstanceDispatcherImpl.java:157)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.dispatchOneInstance(FragmentInstanceDispatcherImpl.java:138)
at org.apache.iotdb.db.mpp.plan.scheduler.FragmentInstanceDispatcherImpl.lambda$dispatchWrite$1(FragmentInstanceDispatcherImpl.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

复现流程：
1. 机器信息
192.168.130.3/4/5
16C32G
2. benchmark 配置见附件
使用的bm版本：/home/benchmark/bm_20220322
3. iotdb配置文件见附件

Issue Links

• links to: GitHub Pull Request #6418