[IOTDB-5015] [write]when writing for about 6 hours to only 1 sensor, the writing stopped with error: too many requests need to process

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0-SNAPSHOT
• Component/s: Core/Server
• Sprint: 2022-11-Cluster
• Assignee: Jinrui Zhang
• Reporter: changxue
• Created: 22/Nov/22 02:28
• Updated: 01/Dec/22 01:18
• Resolved: 25/Nov/22 06:41

Description

[write]when writing for about 6 hours to only 1 sensor, the writing stopped with error: too many requests need to process

environment:
3C3D cluster, 2 replicas

RegionId	Type	Status	Database	SeriesSlotId	TimeSlotId	DataNodeId	Host	RpcPort	Role
10	SchemaRegion	Running	root.aggr.g_0	1	0	1	172.20.70.44	6667	Follower
10	SchemaRegion	Running	root.aggr.g_0	1	0	5	172.20.70.46	6667	Leader
11	DataRegion	Running	root.aggr.g_0	1	10	1	172.20.70.44	6667	Follower
11	DataRegion	Running	root.aggr.g_0	1	10	5	172.20.70.46	6667	Leader

reproduction:
1. start the cluster successfully
2. start the 0.13 benchmark at about 20:23 Nov.21, the benchmark configuration see the attachment of config.properties
3. errors occurred at about 3:20 Nov.22 and writing can't be continued.
4. 6:30 Nov.22, I start benchmark again, but can't write to iotdb successfully
5. run stop-datanode.sh and start-datanode.sh at the bad node of 46
6. start benchmark again, now it can write successfully

172.20.70.46 datanode:

 
2022-11-22 03:20:23,586 [pool-8-IoTDB-WAL-Delete-1] INFO  o.a.i.d.w.n.WALNode$DeleteOutdatedFileTask:367 - WAL node-root.aggr.g_0-11 flushes memTable-4510 to TsFile /data/iotdb/apache-iotdb-0.14.0-SNAPSHOT-all-bin/data/datanode/data/sequence/root.aggr.g_0/11/52/1669036247165-4504-0-0.tsfile, memTable size is 1531600.
2022-11-22 03:20:42,915 [pool-25-IoTDB-ClientRPC-Processor-2$20221121_192013_20413_5.1.0] ERROR o.a.i.d.m.p.s.FragmentInstanceDispatcherImpl:234 - write locally failed. TSStatus: TSStatus(code:606, message:Reject write because there are too many requests need to process), message: Reject write because there are too many requests need to process
2022-11-22 03:20:42,978 [pool-25-IoTDB-ClientRPC-Processor-2$20221121_192043_20414_5.1.0] INFO  o.a.i.c.m.MultiLeaderServerImpl:178 - [Throttle Down] index:380448, safeIndex:380448
2022-11-22 03:20:43,594 [pool-8-IoTDB-WAL-Delete-1] INFO  o.a.i.d.w.n.WALNode$DeleteOutdatedFileTask:242 - Effective information ratio 1.8484028968067935E-4 (active memTables cost is 13563200, flushed memTables cost is 73364378500) of wal node-root.aggr.g_0-11 is below wal min effective info ratio 0.1, some memTables will be snapshot or flushed.

3:20 monitor: