[OAK-3264] Deadlock between persisted compaction map and cleanup 2

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.5, 1.4
• Component/s: segmentmk
• Labels: cleanup compaction gc
• Epic Link: SegmentMK revision GC
• Assignee: Michael Dürig
• Reporter: Michael Dürig
• Created: 21/Aug/15 11:30
• Updated: 16/Feb/21 09:52
• Resolved: 21/Aug/15 12:23

Description

Just seen this deadlock while running SegmentCompactionIT:

"pool-1-thread-47":
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:910)
	- waiting to lock <0x0000000700110bd0> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:211)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
	- locked <0x0000000700328b88> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:154)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:121)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:103)
	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.get(CompactionMap.java:93)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.uncompact(SegmentWriter.java:1074)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1098)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:85)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:214)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:81)
	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.remove(MemoryNodeBuilder.java:355)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.modify(SegmentCompactionIT.java:448)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:430)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:406)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
"TarMK flush thread [target/SegmentCompactionIT9065337410200765612dir], active since Fri Aug 21 06:53:18 GMT+00:00 2015, previous max duration 40846ms":
	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:145)
	- waiting to lock <0x0000000700328b88> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:154)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.compress(PersistedCompactionMap.java:204)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.remove(PersistedCompactionMap.java:155)
	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.remove(CompactionMap.java:108)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:699)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:628)
	- locked <0x0000000700110bd0> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
	- locked <0x000000070017f1c0> (a java.util.concurrent.atomic.AtomicReference)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$1.run(FileStore.java:413)
	at java.lang.Thread.run(Thread.java:745)
	at org.apache.jackrabbit.oak.plugins.segment.file.BackgroundThread.run(BackgroundThread.java:70)

Issue Links

• depends upon: OAK-2862 CompactionMap#compress() inefficient for large compaction maps
• relates to: OAK-3179 Deadlock between persisted compaction map and cleanup