Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-3179

Deadlock between persisted compaction map and cleanup

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Just seen this deadlock while running SegmentCompactionIT:

      "TarMK flush thread [target/SegmentCompactionIT3250704011919039778dir], active since Wed Aug 05 09:25:57 GMT+00:00 2015, previous max duration 2325ms" daemon prio=10 tid=0x00007f5674872800 nid=0x5dc8 waiting for monitor entry [0x00007f5666a00000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:145)
      	- waiting to lock <0x0000000707fc7fe8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
      	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:154)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
      	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.compress(PersistedCompactionMap.java:204)
      	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.remove(PersistedCompactionMap.java:155)
      	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.remove(CompactionMap.java:108)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.cleanup(FileStore.java:694)
      	- locked <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:628)
      	- locked <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	- locked <0x00000007000aed60> (a java.util.concurrent.atomic.AtomicReference)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore$1.run(FileStore.java:413)
      	at java.lang.Thread.run(Thread.java:745)
      	at org.apache.jackrabbit.oak.plugins.segment.file.BackgroundThread.run(BackgroundThread.java:70)
      
      "pool-1-thread-34" prio=10 tid=0x00007f55ec002800 nid=0x5dea waiting for monitor entry [0x00007f56648de000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:904)
      	- waiting to lock <0x000000070017b330> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.readSegment(SegmentTracker.java:210)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:149)
      	- locked <0x0000000707fc7fe8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
      	at org.apache.jackrabbit.oak.plugins.segment.Segment.readString(Segment.java:400)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:215)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.getEntry(MapRecord.java:186)
      	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:121)
      	at org.apache.jackrabbit.oak.plugins.segment.PersistedCompactionMap.get(PersistedCompactionMap.java:103)
      	at org.apache.jackrabbit.oak.plugins.segment.CompactionMap.get(CompactionMap.java:93)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.uncompact(SegmentWriter.java:1074)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1098)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1154)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1135)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:399)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1126)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:100)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore$Commit.<init>(SegmentNodeStore.java:418)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStore.merge(SegmentNodeStore.java:204)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:433)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentCompactionIT$RandomWriter.call(SegmentCompactionIT.java:406)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mduerig Michael Dürig
            mduerig Michael Dürig
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment