Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-8006

SegmentBlob#readLongBlobId might cause SegmentNotFoundException on standby

    XMLWordPrintableJSON

    Details

      Description

      When persisting a segment transferred from master, among others, the cold standby needs to read the binary references from the segment. While this usually doesn't involve any additional reads from any other segments, there is a special case concerning binary IDs larger than 4092 bytes. These can live in other segments (which got transferred prior to the current segment and are already on the standby), but it might also be the case that the binary ID is stored in the same segment. If this happens, the call to blobId.getSegment()[0], triggers a new read of the current, un-persisted segment . Thus, a SegmentNotFoundException is thrown:

      22.01.2019 09:35:59.345 *ERROR* [standby-run-1] org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed synchronizing state.
      org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment d40a9da6-06a2-4dc0-ab91-5554a33c02b0 not found
              at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:284) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:498) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache.get(LocalCache.java:3932) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
              at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:498) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:98) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(SegmentBlob.java:206) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(SegmentBlob.java:163) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.file.AbstractFileStore$3.consume(AbstractFileStore.java:262) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.Segment.forEachRecord(Segment.java:601) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readBinaryReferences(AbstractFileStore.java:257) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(FileStore.java:533) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentFromPrimary(StandbyClientSyncExecution.java:225) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:194) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:76) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:165) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
              at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:347) [org.apache.sling.commons.scheduler:2.7.2]
              at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.2]
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:834)

       

      [0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java#L205

        Attachments

        1. OAK-8006.patch
          1 kB
          Andrei Dulceanu
        2. OAK-8006-02.patch
          7 kB
          Andrei Dulceanu
        3. OAK-8006-test.patch
          6 kB
          Andrei Dulceanu

          Issue Links

            Activity

              People

              • Assignee:
                dulceanu Andrei Dulceanu
                Reporter:
                dulceanu Andrei Dulceanu
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: