Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2560

SegmentStore deadlock on shutdown

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • segmentmk

    Description

      We've ran into a deadlock in the Oak SegmentNodeStore between
      org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300) and org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)
      This was seen on Oak 1.0.11.

      Java stack information for the threads listed above 
       
      "pool-9-thread-5":
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:656)
      	- waiting to lock <0x0000000680024468> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
      	- locked <0x00000006b0b7ae28> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
      	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:128)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:462)
      	- locked <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:460)
      	- locked <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:660)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1020)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1022)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:91)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:77)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:204)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:73)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:331)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:323)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.child(MemoryNodeBuilder.java:311)
      	at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.insert(ContentMirrorStoreStrategy.java:112)
      	at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:81)
      	at org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:261)
      	at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74)
      	at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:130)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeAdded(MapRecord.java:383)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeAdded(MapRecord.java:425)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:483)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
      	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:354)
      	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:300)
      	- locked <0x000000068afa00e8> (a org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
      	at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:105)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
       "FelixStartLevel":
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:167)
      	- waiting to lock <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:431)
      	- locked <0x000000068461ecb8> (a java.util.concurrent.atomic.AtomicReference)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.close(FileStore.java:582)
      	- locked <0x0000000680024468> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300)
      	- locked <0x00000006800243c8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
      	at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
      	at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
      	at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
      	at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.disposeImplementationObject(SingleComponentManager.java:355)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.deleteComponent(SingleComponentManager.java:170)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:908)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:883)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:580)
      	at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:406)
      	at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:335)
      	at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:313)
      	at org.apache.felix.scr.impl.Activator.access$300(Activator.java:45)
      	at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:198)
      	at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
      	at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
      	at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:869)
      	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:790)
      	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:515)
      	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4409)
      	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2526)
      	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1315)
      	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
      	at java.lang.Thread.run(Thread.java:744)
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stillalex Alex Deparvu
            stillalex Alex Deparvu
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment