[OAK-9847] Offline compaction cleanup failures for migrated AzureSegmentStores with missing graph files

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.44.0
• Fix Version/s: 1.46.0
• Component/s: segment-azure, segment-tar
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 19/Jul/22 09:30
• Updated: 21/Dec/22 03:39
• Resolved: 19/Jul/22 15:11

Description

There are cases in which the cleanup step in offline compaction fails for AzureSegmentStores migrated to their Tar counterpart and then compacted. This happens because some AzureSegmentStores are not correctly closed and as a result the graph file is missing for some archives. Moreover, it was observed that some stores contained old archives with a GCGeneration greater than that of the current head, making them (and the segments inside) unreclaimable.

This improvement makes sure that all data segments coming from a generation greater than that of the root record are marked as collectable.

Below an example stacktrace of the cleanup failure:

INFO  [2022-06-13 03:43:04,814] org.apache.jackrabbit.oak.segment.file.FileStore: TarMK GC #0: cleanup started using reclaimer (full generation older than 1.1, with 1 retained generations)
WARN  [2022-06-13 03:43:08,910] org.apache.jackrabbit.oak.segment.file.tar.GraphLoader: Invalid graph magic number
WARN  [2022-06-13 03:43:13,498] org.apache.jackrabbit.oak.segment.file.tar.GraphLoader: Invalid graph magic number
WARN  [2022-06-13 03:43:14,500] org.apache.jackrabbit.oak.segment.file.tar.GraphLoader: Invalid graph magic number
WARN  [2022-06-13 03:43:16,608] org.apache.jackrabbit.oak.segment.file.tar.GraphLoader: Invalid graph magic number
WARN  [2022-06-13 03:43:16,748] org.apache.jackrabbit.oak.segment.file.tar.GraphLoader: Invalid graph magic number
INFO  [2022-06-13 03:43:17,124] org.apache.jackrabbit.oak.segment.file.FileStore: TarMK closed: /mnt/sandbox/tmp/1655089520931-0
java.lang.NullPointerException
	at org.apache.jackrabbit.oak.segment.file.tar.TarReader.getReferences(TarReader.java:328)
	at org.apache.jackrabbit.oak.segment.file.tar.TarReader.mark(TarReader.java:417)
	at org.apache.jackrabbit.oak.segment.file.tar.TarFiles.cleanup(TarFiles.java:690)
	at org.apache.jackrabbit.oak.segment.file.DefaultCleanupStrategy.cleanup(DefaultCleanupStrategy.java:55)
	at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.cleanup(AbstractGarbageCollectionStrategy.java:155)
	at org.apache.jackrabbit.oak.segment.file.AbstractGarbageCollectionStrategy.cleanup(AbstractGarbageCollectionStrategy.java:87)
	at org.apache.jackrabbit.oak.segment.file.SynchronizedGarbageCollectionStrategy.cleanup(SynchronizedGarbageCollectionStrategy.java:73)
	at org.apache.jackrabbit.oak.segment.file.GarbageCollector.cleanup(GarbageCollector.java:305)
	at org.apache.jackrabbit.oak.segment.file.FileStore.cleanup(FileStore.java:419)
	at org.apache.jackrabbit.oak.segment.tool.Compact.run(Compact.java:296)

Issue Links

• links to: GitHub Pull Request #632