Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-2943

[ wal ] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.

    XMLWordPrintableJSON

Details

    • 2022-4-Dragon

    Description

      master 2691ade495513b54f00810c6cdd97c08326293e5
      问题描述:
      benchmark 写入完成,执行flush , stop-server.sh

      的时候,WAL-Sync线程报错,数据库read-only,数据库进程不退出

      2022-04-18 00:23:13,355 [pool-6-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_9-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_9/0/0/1650210199832-211-1-0.inner, status: abort
      2022-04-18 00:28:02,653 [pool-25-IoTDB-WAL-Sync(node-0)-1] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.
      java.io.FileNotFoundException: /data/iotdb_data/wal/0/_4035.wal (No such file or directory)
      at java.io.FileOutputStream.open0(Native Method)
      at java.io.FileOutputStream.open(FileOutputStream.java:270)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      at org.apache.iotdb.db.wal.io.LogWriter.<init>(LogWriter.java:51)
      at org.apache.iotdb.db.wal.io.WALWriter.<init>(WALWriter.java:58)
      at org.apache.iotdb.db.wal.buffer.AbstractWALBuffer.rollLogWriter(AbstractWALBuffer.java:74)
      at org.apache.iotdb.db.wal.buffer.WALBuffer$SyncBufferTask.run(WALBuffer.java:379)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      测试步骤
      1. 机器环境
      8C32G *2
      私有云 iotdb 172.20.70.2
      benchmark 172.20.70.16

      数据库配置
      MAX_HEAP_SIZE="16G"
      wal_buffer_size_in_byte=1048576
      wal_buffer_queue_capacity=1000

      benchmark配置文件见附件。
      benchmark执行完之后,数据库端执行stop_db.sh(见附件)

      数据库的日志:
      2022-04-18 00:23:13,353 [pool-6-IoTDB-Compaction-3] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_6-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, null
      2022-04-18 00:23:13,354 [pool-6-IoTDB-Compaction-9] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_39-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_39/0/0/1650210247564-211-1-0.inner, status: abort
      2022-04-18 00:23:13,355 [pool-6-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.i.InnerSpaceCompactionTask:197 - root.test.g_9-0 [Compaction] Throwable is caught during execution of SizeTieredCompaction, [Compaction] compaction for target file file is /data/iotdb_data/data/sequence/root.test.g_9/0/0/1650210199832-211-1-0.inner, status: abort
      2022-04-18 00:28:02,653 [pool-25-IoTDB-WAL-Sync(node-0)-1] ERROR o.a.i.d.w.b.WALBuffer$SyncBufferTask:382 - Fail to roll wal node-0's log writer, change system mode to read-only.
      java.io.FileNotFoundException: /data/iotdb_data/wal/0/_4035.wal (No such file or directory)
      at java.io.FileOutputStream.open0(Native Method)
      at java.io.FileOutputStream.open(FileOutputStream.java:270)
      at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
      at org.apache.iotdb.db.wal.io.LogWriter.<init>(LogWriter.java:51)
      at org.apache.iotdb.db.wal.io.WALWriter.<init>(WALWriter.java:58)
      at org.apache.iotdb.db.wal.buffer.AbstractWALBuffer.rollLogWriter(AbstractWALBuffer.java:74)
      at org.apache.iotdb.db.wal.buffer.WALBuffer$SyncBufferTask.run(WALBuffer.java:379)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      数据库进程不退出。

      Attachments

        1. config.properties.bk
          14 kB
          刘珍
        2. stop_db.sh
          0.8 kB
          刘珍
        3. log_all.log.gz
          652 kB
          刘珍
        4. log-all-20220417.0.log.gz
          2.62 MB
          刘珍

        Issue Links

          Activity

            People

              ThuLiuxuxin Liuxuxin
              刘珍 刘珍
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: