Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-4675

SNFE thrown while testing FileStore.cleanup() running concurrently with writes

    XMLWordPrintableJSON

Details

    Description

      SegmentNotFoundException is thrown from time to time in the following scenario: plenty of concurrent writes (each creating a 625 bytes blob) interrupted by a cleanup.

      Stack trace (including some debugging statements added by me):

      Pre cleanup readers: []
Before cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar]
Initial size: 357.4 kB
After cleanup readers: [/Users/dulceanu/work/test-repo/data00000a.tar]
After cleanup size: 357.4 kB
Final size: 361.0 kB
Exception in thread "pool-5-thread-74" org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Cannot copy record from a generation that has been gc'ed already
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1207)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1096)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNodeUncached(SegmentWriter.java:1074)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:1013)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.writeNode(SegmentWriter.java:987)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.access$700(SegmentWriter.java:379)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$8.execute(SegmentWriter.java:337)
	at org.apache.jackrabbit.oak.segment.SegmentBufferWriterPool.execute(SegmentBufferWriterPool.java:105)
	at org.apache.jackrabbit.oak.segment.SegmentWriter.writeNode(SegmentWriter.java:334)
	at org.apache.jackrabbit.oak.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:111)
	at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.prepare(SegmentNodeStore.java:550)
	at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.optimisticMerge(SegmentNodeStore.java:571)
	at org.apache.jackrabbit.oak.segment.SegmentNodeStore$Commit.execute(SegmentNodeStore.java:627)
	at org.apache.jackrabbit.oak.segment.SegmentNodeStore.merge(SegmentNodeStore.java:287)
	at org.apache.jackrabbit.oak.segment.CompactionAndCleanupIT$1.run(CompactionAndCleanupIT.java:961)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment 4fb637cc-5013-4925-ab13-0629c4406481 not found
	at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1341)
	at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:123)
	at org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:94)
	at org.apache.jackrabbit.oak.segment.SegmentWriter$SegmentWriteOperation.isOldGeneration(SegmentWriter.java:1199)
	... 18 more
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Invalid segment format. Dumping segment 4fb637cc-5013-4925-ab13-0629c4406481
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b
00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488
00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686...
00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000
000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000
000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603
000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0...
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

	at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1015)
	at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:972)
	at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:283)
	at org.apache.jackrabbit.oak.segment.SegmentCache.getSegment(SegmentCache.java:92)
	at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:1275)
	... 21 more
Caused by: java.lang.IllegalStateException: Invalid segment format. Dumping segment 4fb637cc-5013-4925-ab13-0629c4406481
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000000F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000100 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000110 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000120 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000130 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000140 39 37 39 31 31 36 30 38 2D 63 31 63 65 2D 34 62 97911608-c1ce-4b
00000150 35 63 2D 61 36 33 37 2D 39 36 61 65 39 34 38 38 5c-a637-96ae9488
00000160 61 37 65 38 2E 30 61 62 34 30 36 38 36 00 00 00 a7e8.0ab40686...
00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001A0 00 00 00 00 30 30 30 30 34 30 30 00 30 30 30 30 ....0000400.0000
000001B0 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 000.0000000.0000
000001C0 30 30 30 31 33 30 30 00 31 32 37 35 34 36 30 33 0001300.12754603
000001D0 37 32 32 00 30 31 32 33 30 37 00 20 30 00 00 00 722.012307. 0...
000001E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000001F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000200 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000220 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000230 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000250 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000260 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000270 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000280 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00000290 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000002A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000002B0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

	at com.google.common.base.Preconditions.checkState(Preconditions.java:150)
	at org.apache.jackrabbit.oak.segment.Segment.<init>(Segment.java:185)
	at org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1292)
	at org.apache.jackrabbit.oak.segment.file.FileStore$15.call(FileStore.java:1)
	at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1011)
	... 25 more
0

      Attachments

        1. OAK-4675-oak-segment-03.patch
          6 kB
          Andrei Dulceanu
        2. OAK-4675-02.patch
          6 kB
          Michael Dürig
        3. OAK-4675-01.patch
          4 kB
          Andrei Dulceanu
        4. test-case.patch
          82 kB
          Andrei Dulceanu

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. OAK-4685 Avoid concurrent calls to FileStore.cleanup() and FileStore.compact()

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. OAK-3753 Test failure: HeavyWriteIT

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              adulceanu Andrei Dulceanu
              adulceanu Andrei Dulceanu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: