[IOTDB-2943] [ wal ] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0-SNAPSHOT
• Fix Version/s: 0.14.0
• Component/s: Core/WAL
• Labels: pull-request-available
• Sprint: 2022-4-Dragon
• Assignee: Liuxuxin
• Reporter: 刘珍
• Created: 18/Apr/22 02:05
• Updated: 11/May/22 02:11
• Resolved: 25/Apr/22 01:09

Description

master 2691ade495513b54f00810c6cdd97c08326293e5
问题描述：
benchmark 写入完成，执行flush , stop-server.sh

的时候，WAL-Sync线程报错，数据库read-only，数据库进程不退出。

2022-04-18 00:23:13,355 [pool-6-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_9-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_9/0/0/1650210199832-211-1-0.inner, status: abort
2022-04-18 00:28:02,653 [pool-25-IoTDB-WAL-Sync(node-0)-1] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.
java.io.FileNotFoundException: /data/iotdb_data/wal/0/_4035.wal (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.apache.iotdb.db.wal.io.LogWriter.<init>(LogWriter.java:51)
at org.apache.iotdb.db.wal.io.WALWriter.<init>(WALWriter.java:58)
at org.apache.iotdb.db.wal.buffer.AbstractWALBuffer.rollLogWriter(AbstractWALBuffer.java:74)
at org.apache.iotdb.db.wal.buffer.WALBuffer$SyncBufferTask.run(WALBuffer.java:379)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

测试步骤
1. 机器环境
8C32G *2
私有云 iotdb 172.20.70.2
benchmark 172.20.70.16

数据库配置
MAX_HEAP_SIZE="16G"
wal_buffer_size_in_byte=1048576
wal_buffer_queue_capacity=1000

benchmark配置文件见附件。
benchmark执行完之后，数据库端执行stop_db.sh（见附件）

数据库的日志：
2022-04-18 00:23:13,353 [pool-6-IoTDB-Compaction-3] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_6-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, null
2022-04-18 00:23:13,354 [pool-6-IoTDB-Compaction-9] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_39-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_39/0/0/1650210247564-211-1-0.inner, status: abort
2022-04-18 00:23:13,355 [pool-6-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_9-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_9/0/0/1650210199832-211-1-0.inner, status: abort
2022-04-18 00:28:02,653 [pool-25-IoTDB-WAL-Sync(node-0)-1] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.
java.io.FileNotFoundException: /data/iotdb_data/wal/0/_4035.wal (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at org.apache.iotdb.db.wal.io.LogWriter.<init>(LogWriter.java:51)
at org.apache.iotdb.db.wal.io.WALWriter.<init>(WALWriter.java:58)
at org.apache.iotdb.db.wal.buffer.AbstractWALBuffer.rollLogWriter(AbstractWALBuffer.java:74)
at org.apache.iotdb.db.wal.buffer.WALBuffer$SyncBufferTask.run(WALBuffer.java:379)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

数据库进程不退出。

Issue Links

• links to: GitHub Pull Request #5591