[OAK-10814] Tar file recovery still fails with SNFE in some cases.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.62.0
• Fix Version/s: 1.64.0
• Component/s: segment-tar
• Assignee: Miroslav Smiljanic
• Reporter: Axel Hanikel
• Created: 17/May/24 11:15
• Updated: 27/May/24 19:51
• Resolved: 17/May/24 14:12

Description

13.05.2024 06:59:56.357	WARN	[FelixStartLevel] org.apache.jackrabbit.oak.segment.file.tar.SegmentTarReader Unable to load index of file data21604a.tar: Unrecognized magic number
13.05.2024 06:59:56.369	INFO	[FelixStartLevel] org.apache.jackrabbit.oak.segment.file.tar.TarReader Backing up data21604a.tar to data21604a.tar.bak
13.05.2024 06:59:56.372	INFO	[FelixStartLevel] org.apache.jackrabbit.oak.segment.file.tar.TarReader Regenerating tar file data21604a.tar
13.05.2024 06:59:56.439	ERROR	[FelixStartLevel] org.apache.jackrabbit.oak.segment.SegmentNotFoundExceptionListener Segment not found: a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652. SegmentId age=18ms
org.apache.jackrabbit.oak.segment.SegmentNotFoundException: Segment a4ad6e4b-93dc-4e09-ad0d-ab99a7f93652 not found
	at org.apache.jackrabbit.oak.segment.file.AbstractFileStore.readSegmentUncached(AbstractFileStore.java:297) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.file.FileStore.lambda$readSegment$10(FileStore.java:512) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.lambda$getSegment$0(SegmentCache.java:163) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4938) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3576) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2318) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2191) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$Segment.get(LocalCache.java:2081) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache.get(LocalCache.java:4019) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4933) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentCache$NonEmptyCache.getSegment(SegmentCache.java:160) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.file.FileStore.readSegment(FileStore.java:512) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentId.getSegment(SegmentId.java:153) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.Record.getSegment(Record.java:71) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:99) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.ListRecord.getEntries(ListRecord.java:92) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentStream.read(SegmentStream.java:166) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.guava.common.io.ByteStreams.read(ByteStreams.java:934) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:802) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.guava.common.io.ByteStreams.readFully(ByteStreams.java:786) [org.apache.jackrabbit.guava:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentStream.getString(SegmentStream.java:105) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.Segment.readString(Segment.java:467) [org.apache.jackrabbit.oak-segment-tar:1.62.0]
	at org.apache.jackrabbit.oak.segment.SegmentBlob.readLongBlobId(SegmentBlob.java:248) [org.apache.jackrabbit.oak-segment-tar:1.62.0]

The reason seems to be that the long blobId string is stored behind a list record, where the elements are stored in yet a different segment, which is then loaded in the normal way, i.e. bypassing the recovery map added in OAK-10636.

Issue Links

• relates to: OAK-10636 Tar recovery unsuccessful in oak-segment-tar during initialisation of FileStore
• links to: GitHub Pull Request #1468