[OAK-8006] SegmentBlob#readLongBlobId might cause SegmentNotFoundException on standby

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.12.0, 1.10.1, 1.8.12
• Component/s: segment-tar, tarmk-standby
• Labels: cold-standby
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 28/Jan/19 12:24
• Updated: 04/Oct/19 17:52
• Resolved: 22/Feb/19 10:45

Description

When persisting a segment transferred from master, among others, the cold standby needs to read the binary references from the segment. While this usually doesn't involve any additional reads from any other segments, there is a special case concerning binary IDs larger than 4092 bytes. These can live in other segments (which got transferred prior to the current segment and are already on the standby), but it might also be the case that the binary ID is stored in the same segment. If this happens, the call to blobId.getSegment()[0], triggers a new read of the current, un-persisted segment . Thus, a SegmentNotFoundException is thrown:

22.01.2019 09:35:59.345 *ERROR* [standby-run-1] org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync Failed synchronizing state.
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment d40a9da6-06a2-4dc0-ab91-5554a33c02b0 not found
        at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:284) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:498) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4724) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3932) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4721) [com.adobe.granite.osgi.wrapper.guava:15.0.0.0002]
        at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:498) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.RecordId.getSegment(RecordId.java:98) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(SegmentBlob.java:206) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.SegmentBlob.readBlobId(SegmentBlob.java:163) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.file.AbstractFileStore$3.consume(AbstractFileStore.java:262) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.Segment.forEachRecord(Segment.java:601) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readBinaryReferences(AbstractFileStore.java:257) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.file.FileStore.writeSegment(FileStore.java:533) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentFromPrimary(StandbyClientSyncExecution.java:225) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:194) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:76) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:165) [org.apache.jackrabbit.oak-segment-tar:1.10.0]
        at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:347) [org.apache.sling.commons.scheduler:2.7.2]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.2]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

 

[0] https://github.com/apache/jackrabbit-oak/blob/trunk/oak-segment-tar/src/main/java/org/apache/jackrabbit/oak/segment/SegmentBlob.java#L205

Issue Links

• relates to: OAK-8063 The cold standby client doesn't correctly handle backward references