[OAK-3179] Deadlock between persisted compaction map and cleanup

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.3.4, 1.4
• Component/s: segmentmk
• Labels: cleanup compaction gc
• Epic Link: SegmentMK revision GC
• Assignee: Michael Dürig
• Reporter: Michael Dürig
• Created: 05/Aug/15 10:39
• Updated: 16/Feb/21 09:52
• Resolved: 05/Aug/15 13:27

Description

Just seen this deadlock while running SegmentCompactionIT:

"TarMK flush thread [target/SegmentCompactionIT3250704011919039778dir], active since Wed Aug 05 09:25:57 GMT+00:00 2015, previous max duration 2325ms" daemon prio=10 tid=0x00007f5674872800 nid=0x5dc8 waiting for monitor entry [0x00007f5666a00000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:145)
	- waiting to lock <0x0000000707fc7fe8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:154)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.compress(PersistedCompactionMap.java:204)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.remove(PersistedCompactionMap.java:155)
	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.remove(CompactionMap.java:108)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:694)
	- locked <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:628)
	- locked <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
	- locked <0x00000007000aed60> (a java.util.concurrent.atomic.AtomicReference)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$1.run(FileStore.java:413)
	at java.lang.Thread.run(Thread.java:745)
	at org.apache.jackrabbit.oak.plugins.segment.file.BackgroundThread.run(BackgroundThread.java:70)

"pool-1-thread-34" prio=10 tid=0x00007f55ec002800 nid=0x5dea waiting for monitor entry [0x00007f56648de000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:904)
	- waiting to lock <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:210)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
	- locked <0x0000000707fc7fe8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:400)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:215)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:121)
	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:103)
	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.get(CompactionMap.java:93)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.uncompact(SegmentWriter.java:1074)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1098)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.<init>(SegmentNodeStore.java:418)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:204)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:433)
	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:406)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Issue Links

• depends upon: OAK-2862 CompactionMap#compress() inefficient for large compaction maps
• is related to: OAK-3264 Deadlock between persisted compaction map and cleanup 2