Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.1.0
-
None
-
None
Description
测试版本： iotdb master 0427_ed8d539
日志路径：192.168.130.2 16C32G /data2/i_m_0427_ed8d539/logs
问题描述：
2023-04-27 19:51:27,391 [pool-16-IoTDB-Compaction-1] ERROR o.a.i.d.e.c.e.t.CrossSpaceCompactionTask:251 - root.test.g_0-2 [Compaction] Meet errors in cross space compaction.
java.lang.RuntimeException: java.nio.BufferUnderflowException
at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:152)
at org.apache.iotdb.db.engine.compaction.execute.task.CrossSpaceCompactionTask.doCompaction(CrossSpaceCompactionTask.java:160)
at org.apache.iotdb.db.engine.compaction.execute.task.AbstractCompactionTask.start(AbstractCompactionTask.java:80)
at org.apache.iotdb.db.engine.compaction.schedule.CompactionWorker.run(CompactionWorker.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.BufferUnderflowException: null
at java.nio.Buffer.nextGetIndex(Buffer.java:524)
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:138)
at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:67)
at org.apache.iotdb.tsfile.utils.BitReader.next(BitReader.java:36)
at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.decodeIndex(FreqDecoder.java:129)
at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.loadBlock(FreqDecoder.java:86)
at org.apache.iotdb.tsfile.encoding.decoder.FreqDecoder.readDouble(FreqDecoder.java:45)
at org.apache.iotdb.tsfile.read.reader.page.ValuePageReader.writeColumnBuilderWithNextBatch(ValuePageReader.java:306)
at org.apache.iotdb.tsfile.read.reader.page.AlignedPageReader.getAllSatisfiedData(AlignedPageReader.java:226)
at org.apache.iotdb.tsfile.read.reader.chunk.AlignedChunkReader.readPageData(AlignedChunkReader.java:320)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PageElement.deserializePage(PageElement.java:96)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.element.PointElement.<init>(PointElement.java:35)
at org.apache.iotdb.db.engine.compaction.execute.utils.reader.PointPriorityReader.addNewPage(PointPriorityReader.java:175)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.checkAndCompactOverlapPage(SeriesCompactionExecutor.java:341)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapPages(SeriesCompactionExecutor.java:307)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactPages(SeriesCompactionExecutor.java:236)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactWithOverlapChunks(SeriesCompactionExecutor.java:167)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.SeriesCompactionExecutor.compactChunks(SeriesCompactionExecutor.java:147)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.compactFiles(AlignedSeriesCompactionExecutor.java:97)
at org.apache.iotdb.db.engine.compaction.execute.utils.executor.fast.AlignedSeriesCompactionExecutor.execute(AlignedSeriesCompactionExecutor.java:84)
at org.apache.iotdb.db.engine.compaction.execute.task.subtask.FastCompactionPerformerSubTask.call(FastCompactionPerformerSubTask.java:141)
at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.compactAlignedSeries(FastCompactionPerformer.java:199)
at org.apache.iotdb.db.engine.compaction.execute.performer.impl.FastCompactionPerformer.perform(FastCompactionPerformer.java:133)
... 8 common frames omitted
测试流程
1. start-standalone.sh 启动1C1D
confignode-env.sh
MAX_HEAP_SIZE="2G"
datanode-env.sh
MAX_HEAP_SIZE="16G"
MAX_DIRECT_MEMORY_SIZE="4G"
iotdb-common.properties
series_slot_num=1000
default_storage_group_level=2
2.创建2个dev，12序列/dev，DOUBLE类型，指定不同的压缩方式
不同编码：
dev定义如下：
create database root.test.g_0;
create ALIGNED timeseries root.test.g_0.d_0(s_0 DOUBLE compressor=SNAPPY,s_1 DOUBLE compressor=LZ4,s_2 DOUBLE compressor=GZIP,s_3 DOUBLE compressor=ZSTD,s_4 DOUBLE compressor=LZMA2,s_5 DOUBLE compressor=UNCOMPRESSED,s_6 DOUBLE compressor=LZ4,s_7 DOUBLE compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE compressor=LZMA2,s_10 DOUBLE compressor=SNAPPY,s_11 DOUBLE compressor=UNCOMPRESSED);
create ALIGNED timeseries root.test.g_0.d_1(s_0 DOUBLE compressor=SNAPPY,s_1 DOUBLE ENCODING=RLE compressor=LZ4,s_2 DOUBLE ENCODING=TS_2DIFF compressor=GZIP,s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,s_4 DOUBLE ENCODING=FREQ compressor=LZMA2,s_5 DOUBLE ENCODING=CHIMP compressor=UNCOMPRESSED,s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,s_7 DOUBLE ENCODING=RLBE compressor=GZIP,s_8 DOUBLE compressor=ZSTD,s_9 DOUBLE compressor=LZMA2,s_10 DOUBLE compressor=SNAPPY,s_11 DOUBLE compressor=UNCOMPRESSED)
每个序列写入960个点。见insert.sh 先写d_0，再写d_1,需要编辑一下脚本中的dev名字。
3. 再创建998个dev，用bm写入数据
运行如下脚本，创建dev
#!/bin/bash
for i in
do
./sbin/start-cli.sh -e "create ALIGNED timeseries root.test.g_0.d_$i(s_0 DOUBLE compressor=SNAPPY,\
s_1 DOUBLE ENCODING=RLE compressor=LZ4,\
s_2 DOUBLE ENCODING=TS_2DIFF compressor=GZIP,\
s_3 DOUBLE ENCODING=GORILLA compressor=ZSTD,\
s_4 DOUBLE ENCODING=FREQ compressor=LZMA2,\
s_5 DOUBLE ENCODING=CHIMP compressor=UNCOMPRESSED,\
s_6 DOUBLE ENCODING=SPRINTZ compressor=LZ4,\
s_7 DOUBLE ENCODING=RLBE compressor=GZIP,\
s_8 DOUBLE compressor=ZSTD,\
s_9 DOUBLE compressor=LZMA2,\
s_10 DOUBLE compressor=SNAPPY,\
s_11 DOUBLE compressor=UNCOMPRESSED) "
done
运行附件中的配置，写入数据：
先运行：0427_aligned_compress.conf
再运行：0427_aligned_compress_unseq.conf
大概2小时30分钟后，出现上述异常。
