Uploaded image for project: 'Apache IoTDB'
  1. Apache IoTDB
  2. IOTDB-714

[No disk space]Can not restart IoTDB after disk space cleanup

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      Hi IoTDB Developers:

      I got an issue when I using iotdb-master[0.10.0-snapshot] to run aging test.After almost 7 days[152Hours] there is not enough disk space available for this operation. So we got this log:
      2020-05-25 00:09:12,933 [pool-7-IoTDB-RPC-Client-thread-53] ERROR org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode:226 - Log node root.group_11-1590335956489-7-0.tsfile sync failed, change system mode to read-only
      java.io.IOException: No space left on device
      at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
      at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
      at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
      at sun.nio.ch.IOUtil.write(IOUtil.java:65)
      at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
      at org.apache.iotdb.db.writelog.io.LogWriter.write(LogWriter.java:78)
      at org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.sync(ExclusiveWriteLogNode.java:224)
      at org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.putLog(ExclusiveWriteLogNode.java:106)
      at org.apache.iotdb.db.writelog.node.ExclusiveWriteLogNode.write(ExclusiveWriteLogNode.java:86)
      at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.insertTablet(TsFileProcessor.java:203)
      at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTabletToTsFileProcessor(StorageGroupProcessor.java:730)
      at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.insertTablet(StorageGroupProcessor.java:681)
      at org.apache.iotdb.db.engine.StorageEngine.insertTablet(StorageEngine.java:279)
      at org.apache.iotdb.db.qp.executor.PlanExecutor.insertTablet(PlanExecutor.java:969)
      at org.apache.iotdb.db.service.TSServiceImpl.insertTablet(TSServiceImpl.java:1171)
      at org.apache.iotdb.service.rpc.thrift.TSIService$Processor$insertTablet.getResult(TSIService.java:21982020-05-25 00:09:14,769 [pool-1-IoTDB-Flush-ServerServiceImpl-thread-27] ERROR org.apache.iotdb.db.engine.storagegroup.TsFileProcessor:542 - root.group_17: 1590336165770-7-0.tsfile meet error when flushing a memtable, change system mode to read-only
      java.util.concurrent.ExecutionException: org.apache.iotdb.db.exception.runtime.FlushRunTimeException: java.io.IOException: No space left on device

      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      at org.apache.iotdb.db.engine.flush.MemTableFlushTask.syncFlushMemTable(MemTableFlushTask.java:109)
      at org.apache.iotdb.db.engine.storagegroup.TsFileProcessor.flushOneMemTable(TsFileProcessor.java:540)
      at org.apache.iotdb.db.engine.flush.FlushManager$FlushThread.runMayThrow(FlushManager.java:91)
      at org.apache.iotdb.db.concurrent.WrappedRunnable.run(WrappedRunnable.java:32)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.iotdb.db.exception.runtime.FlushRunTimeException: java.io.IOException: No space left on device
      at org.apache.iotdb.db.engine.flush.MemTableFlushTask.lambda$new$0(MemTableFlushTask.java:243)
      ... 5 common frames omitted
      Caused by: java.io.IOException: No space left on device
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at org.apache.iotdb.tsfile.write.writer.LocalTsFileOutput.getPosition(LocalTsFileOutput.java:61)
      at org.apache.iotdb.tsfile.write.writer.TsFileIOWriter.getPos(TsFileIOWriter.java:325)
      at org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl.writeAllPagesOfChunkToTsFile(ChunkWriterImpl.java:311)
      at org.apache.iotdb.tsfile.write.chunk.ChunkWriterImpl.writeToFileWriter(ChunkWriterImpl.java:220)
      at org.apache.iotdb.db.engine.flush.MemTableFlushTask.lambda$new$0(MemTableFlushTask.java:236)
      ... 5 common frames omitted

      At this time I can use start-cli.sh to connect to server. I tried to use stop-server.sh to stop iotdb service,but it's failed.I saw it when I using command 'jps' .

      After I deleted some useless file in this server,I restart the server.Then I tried to restart iotdb.But I got this error:

      2020-05-26 10:18:39,463 [pool-6-IoTDB-Recovery-Thread-Pool-thread-1] INFO org.apache.iotdb.tsfile.read.TsFileSequenceReader:921 - TsFile /data/fengqingxin/incubator-iotdb_local/server/target/iotdb-server-0.10.0-SNAPSHOT/sbin/./../data/data/sequence/root.group_9/2549/1590335934313-7-0.tsfile self-check cannot proceed at position 526831142 recovered, because : reach the end of the file.

      2020-05-26 10:18:39,818 [main] ERROR org.apache.iotdb.db.concurrent.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread main-1
      java.lang.ExceptionInInitializerError: null
      at org.apache.iotdb.db.engine.StorageEngine.getInstance(StorageEngine.java:103)
      at org.apache.iotdb.db.service.UpgradeSevice.countUpgradeFiles(UpgradeSevice.java:107)
      at org.apache.iotdb.db.service.UpgradeSevice.start(UpgradeSevice.java:60)
      at org.apache.iotdb.db.service.RegisterManager.register(RegisterManager.java:48)
      at org.apache.iotdb.db.service.IoTDB.setUp(IoTDB.java:100)
      at org.apache.iotdb.db.service.IoTDB.active(IoTDB.java:72)
      at org.apache.iotdb.db.service.IoTDB.main(IoTDB.java:58)
      Caused by: org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
      at org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:164)
      at org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:73)
      at org.apache.iotdb.db.engine.StorageEngine$InstanceHolder.<clinit>(StorageEngine.java:99)
      ... 7 common frames omitted
      Caused by: java.util.concurrent.ExecutionException: java.nio.BufferUnderflowException
      at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      at org.apache.iotdb.db.engine.StorageEngine.<init>(StorageEngine.java:162)
      ... 9 common frames omitted
      Caused by: java.nio.BufferUnderflowException: null
      at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:151)
      at org.apache.iotdb.tsfile.utils.ReadWriteIOUtils.readStringWithLength(ReadWriteIOUtils.java:529)
      at org.apache.iotdb.db.qp.physical.PhysicalPlan.readString(PhysicalPlan.java:125)
      at org.apache.iotdb.db.qp.physical.crud.InsertTabletPlan.deserializeFrom(InsertTabletPlan.java:321)
      at org.apache.iotdb.db.qp.physical.PhysicalPlan$Factory.create(PhysicalPlan.java:153)
      at org.apache.iotdb.db.writelog.io.BatchLogReader.readLogs(BatchLogReader.java:52)
      at org.apache.iotdb.db.writelog.io.BatchLogReader.<init>(BatchLogReader.java:44)
      at org.apache.iotdb.db.writelog.io.SingleFileLogReader.hasNext(SingleFileLogReader.java:96)
      at org.apache.iotdb.db.writelog.io.MultiFileLogReader.hasNext(MultiFileLogReader.java:61)
      at org.apache.iotdb.db.writelog.recover.LogReplayer.replayLogs(LogReplayer.java:91)

      at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.redoLogs(TsFileRecoverPerformer.java:196)
      at org.apache.iotdb.db.writelog.recover.TsFileRecoverPerformer.recover(TsFileRecoverPerformer.java:133)
      at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recoverSeqFiles(StorageGroupProcessor.java:516)
      at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.recover(StorageGroupProcessor.java:283)
      at org.apache.iotdb.db.engine.storagegroup.StorageGroupProcessor.<init>(StorageGroupProcessor.java:263)
      at org.apache.iotdb.db.engine.StorageEngine.lambda$new$0(StorageEngine.java:152)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      IoTDB can not be started again!

      Please help me to check this.

      Thanks!

      commit edea3a4c5d066cca16dba3797e9c0ce92d390345

        Attachments

        1. logs.tar.gz
          50.38 MB
          FengQingxin

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                QX.Feng FengQingxin
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: