[OAK-2294] Corrupt repository after concurrent version operations

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.12, 1.1.7
• Component/s: segmentmk
• Labels: corruption
• Assignee: Alex Deparvu
• Reporter: Michael Dürig
• Created: 25/Nov/14 17:08
• Updated: 08/Oct/19 15:21
• Resolved: 20/Feb/15 20:51

Description

Performing version operations (checkin / checkout / addVersionLabel) concurrently can corrupt the repository.

Executing the following code in parallel from multiple threads demonstrates this:

Version version = versionManager.checkin(vPath);
versionManager.checkout(vPath);
String label = version.getName() + " " + Thread.currentThread().getName();
version.getContainingHistory()
    .addVersionLabel(version.getName(), label, true);

In my tests this eventually lead to all sorts of exceptions:

java.lang.IllegalStateException: RefId '85' doesn't exist in data segment 0c5c0814-902c-429c-ad41-cd82aea276a2
	at org.apache.jackrabbit.oak.plugins.segment.Segment.getRefId(Segment.java:196)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.internalReadRecordId(Segment.java:307)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readRecordId(Segment.java:303)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getBucketList(MapRecord.java:134)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntries(MapRecord.java:347)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntries(MapRecord.java:325)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:474)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:394)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:544)
...

java.lang.IllegalStateException: String is too long: 2159501163930351661
	at org.apache.jackrabbit.oak.plugins.segment.Segment.loadString(Segment.java:352)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:319)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:313)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.loadTemplate(Segment.java:418)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:367)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:361)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:78)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:408)
...

java.lang.IllegalStateException
	at com.google.common.base.Preconditions.checkState(Preconditions.java:134)
	at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.writeEntry(TarWriter.java:206)
	at org.apache.jackrabbit.oak.plugins.segment.file.TarWriter.writeEntry(TarWriter.java:200)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.writeSegment(FileStore.java:682)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:228)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.prepare(SegmentWriter.java:329)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeTemplate(SegmentWriter.java:969)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1039)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1062)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:395)
...

Caused by: java.lang.IllegalArgumentException: Invalid type tag: 81
	at org.apache.jackrabbit.oak.api.Type.fromTag(Type.java:202)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.loadTemplate(Segment.java:418)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:367)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readTemplate(Segment.java:361)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getTemplate(SegmentNodeState.java:78)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.getProperty(SegmentNodeState.java:122)
...

Caused by: java.lang.IllegalStateException
	at com.google.common.base.Preconditions.checkState(Preconditions.java:134)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.pos(Segment.java:178)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.loadString(Segment.java:326)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:319)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:313)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.getValue(SegmentPropertyState.java:174)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentPropertyState.getValue(SegmentPropertyState.java:147)
	at org.apache.jackrabbit.oak.plugins.memory.AbstractPropertyState.equal(AbstractPropertyState.java:53)
...

Will attach a patch with a test case shortly.

Issue Links

• is blocked by: OAK-2527 Update SegmentMK header format definition
• is depended upon by: OAK-2481 IllegalStateException in TarMk with large number of properties
• is related to: OAK-2545 SegmentMk IT tests are too intensive
• relates to: OAK-2498 Root record references provide too little context for parsing a segment