Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.0.32, 1.2.18, 1.4.6, Segment Tar 0.0.8
Description
SegmentNotFoundException is thrown from time to time in the following scenario: plenty of concurrent writes (each creating a 625 bytes blob) interrupted by a cleanup.
Stack trace (including some debugging statements added by me):
Pre cleanup readers: [] Before cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar] Initial size: 357.4 kB After cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar] After cleanup size: 357.4 kB Final size: 361.0 kB Exception in thread "pool-5-thread-74" org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy record from a generation that has been gc'ed already at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379) at org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337) at org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105) at org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334) at org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111) at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550) at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571) at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627) at org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287) at org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341) at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123) at org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94) at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199) ... 18 more Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Invalid segment format. Dumping segment 4fb637cc-5013-4925-ab13-0629c4406481 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b 00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488 00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686... 00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000 000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000 000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603 000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0... 000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1015) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:972) at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:283) at org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92) at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1275) ... 21 more Caused by: java.lang.IllegalStateException: Invalid segment format. Dumping segment 4fb637cc-5013-4925-ab13-0629c4406481 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b 00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488 00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686... 00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000 000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000 000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603 000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0... 000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ at com.google.common.base.Preconditions.checkState(Preconditions.java:150) at org.apache.jackrabbit.oak.segment.Segment.<init>(Segment.java:185) at org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1292) at org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1011) ... 25 more 0