[IOTDB-2741] [IDTable] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0-SNAPSHOT
• Component/s: Core/Schema Manager
• Assignee: Kaifeng Xue
• Reporter: 刘珍
• Created: 15/Mar/22 05:45
• Updated: 11/May/22 02:14
• Resolved: 16/Mar/22 11:48

Description

测试版本：
https://github.com/apache/iotdb/pull/5214

问题描述:
开启id table，benchmark工具写入，写入完成立即kill -u 当前用户，备份iotdb的数据，
重启iotdb恢复，报错：
2022-03-15 13:31:56,249 [pool-9-IoTDB-Recovery-Thread-Pool-5] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,249 [pool-9-IoTDB-Recovery-Thread-Pool-5] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,251 [pool-9-IoTDB-Recovery-Thread-Pool-2] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,252 [pool-9-IoTDB-Recovery-Thread-Pool-2] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,253 [pool-9-IoTDB-Recovery-Thread-Pool-1] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,253 [pool-9-IoTDB-Recovery-Thread-Pool-1] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,254 [pool-9-IoTDB-Recovery-Thread-Pool-3] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,254 [pool-9-IoTDB-Recovery-Thread-Pool-3] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,255 [Thread-2] ERROR o.a.i.d.c.IoTDBDefaultThreadExceptionHandler:31 - Exception in thread Thread-2-25
org.apache.iotdb.db.exception.runtime.StorageEngineFailureException: StorageEngine failed to recover.
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:245)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: can't initialize disk schema manager at ./sbin/../data/system/storage_groups/root.test.g_30/SeriesKeyMapping.meta
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.iotdb.db.engine.StorageEngine.lambda$recover$1(StorageEngine.java:243)
... 1 common frames omitted
Caused by: java.lang.IllegalArgumentException: can't initialize disk schema manager at ./sbin/../data/system/storage_groups/root.test.g_30/SeriesKeyMapping.meta
at org.apache.iotdb.db.metadata.idtable.AppendOnlyDiskSchemaManager.<init>(AppendOnlyDiskSchemaManager.java:71)
at org.apache.iotdb.db.metadata.idtable.IDTableHashmapImpl.<init>(IDTableHashmapImpl.java:82)
at org.apache.iotdb.db.metadata.idtable.IDTableManager.lambda$getIDTable$0(IDTableManager.java:90)
at java.util.HashMap.computeIfAbsent(HashMap.java:1127)
at org.apache.iotdb.db.metadata.idtable.IDTableManager.getIDTable(IDTableManager.java:86)
at org.apache.iotdb.db.engine.storagegroup.VirtualStorageGroupProcessor.<init>(VirtualStorageGroupProcessor.java:398)
at org.apache.iotdb.db.engine.StorageEngine.buildNewStorageGroupProcessor(StorageEngine.java:552)
at org.apache.iotdb.db.engine.storagegroup.virtualSg.StorageGroupManager.lambda$asyncRecover$0(StorageGroupManager.java:208)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 common frames omitted
2022-03-15 13:31:56,256 [pool-9-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,256 [pool-9-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,257 [pool-9-IoTDB-Recovery-Thread-Pool-7] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,258 [pool-9-IoTDB-Recovery-Thread-Pool-7] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,259 [pool-9-IoTDB-Recovery-Thread-Pool-8] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,259 [pool-9-IoTDB-Recovery-Thread-Pool-8] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,260 [pool-9-IoTDB-Recovery-Thread-Pool-6] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,260 [pool-9-IoTDB-Recovery-Thread-Pool-6] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,261 [pool-9-IoTDB-Recovery-Thread-Pool-8] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,262 [pool-9-IoTDB-Recovery-Thread-Pool-8] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,262 [pool-9-IoTDB-Recovery-Thread-Pool-7] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,263 [pool-9-IoTDB-Recovery-Thread-Pool-7] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,264 [pool-9-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,264 [pool-9-IoTDB-Recovery-Thread-Pool-4] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,267 [pool-9-IoTDB-Recovery-Thread-Pool-3] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,267 [pool-9-IoTDB-Recovery-Thread-Pool-3] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption
2022-03-15 13:31:56,268 [pool-9-IoTDB-Recovery-Thread-Pool-1] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:136 - File length isn't right
2022-03-15 13:31:56,268 [pool-9-IoTDB-Recovery-Thread-Pool-1] ERROR o.a.i.d.m.i.AppendOnlyDiskSchemaManager:70 - File corruption

测试过程：
1. 测试机器
私有云172.20.70.4
iotdb路径
/data/iotdb/0314_id_table_pr
benchmark路径
/data/iotdb/bm_0314

配置文件见附件
device_id_transformation_method=SHA256
enable_id_table=true
enable_id_table_log_file=true

2. 启动iotdb

3. benchmark写入
配置文件见附件。

benchmark执行完之后，立即执行kill -u cluster

4. 备份data (用于bug复现）
备份路径：
/data/iotdb/0314_id_table_pr/data_id_table_kill-iotdb

5. 清缓存
suod -s<<EOF
echo 3 >/proc/sys/vm/drop_caches
EOF

6.启动iotdb

报错，error日志见附件。