Uploaded image for project: 'Jackrabbit Oak'
  1. Jackrabbit Oak
  2. OAK-2560

SegmentStore deadlock on shutdown

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: segmentmk
    • Labels:

      Description

      We've ran into a deadlock in the Oak SegmentNodeStore between
      org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300) and org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)
      This was seen on Oak 1.0.11.

      Java stack information for the threads listed above 
       
      "pool-9-thread-5":
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.readSegment(FileStore.java:656)
      	- waiting to lock <0x0000000680024468> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentTracker.getSegment(SegmentTracker.java:120)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentId.getSegment(SegmentId.java:101)
      	- locked <0x00000006b0b7ae28> (a org.apache.jackrabbit.oak.plugins.segment.SegmentId)
      	at org.apache.jackrabbit.oak.plugins.segment.Record.getSegment(Record.java:82)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.size(MapRecord.java:128)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:462)
      	- locked <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMapBucket(SegmentWriter.java:460)
      	- locked <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeMap(SegmentWriter.java:660)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1020)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:1022)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter$2.childNodeChanged(SegmentWriter.java:1003)
      	at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:396)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.writeNode(SegmentWriter.java:994)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.getNodeState(SegmentNodeBuilder.java:91)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:77)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.updated(MemoryNodeBuilder.java:204)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeBuilder.updated(SegmentNodeBuilder.java:73)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:331)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.setChildNode(MemoryNodeBuilder.java:323)
      	at org.apache.jackrabbit.oak.plugins.memory.MemoryNodeBuilder.child(MemoryNodeBuilder.java:311)
      	at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.insert(ContentMirrorStoreStrategy.java:112)
      	at org.apache.jackrabbit.oak.plugins.index.property.strategy.ContentMirrorStoreStrategy.update(ContentMirrorStoreStrategy.java:81)
      	at org.apache.jackrabbit.oak.plugins.index.property.PropertyIndexEditor.leave(PropertyIndexEditor.java:261)
      	at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.leave(CompositeEditor.java:74)
      	at org.apache.jackrabbit.oak.spi.commit.VisibleEditor.leave(VisibleEditor.java:63)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:130)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.memory.EmptyNodeState.compareAgainstEmptyState(EmptyNodeState.java:160)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:391)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeAdded(EditorDiff.java:125)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeAdded(MapRecord.java:383)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeAdded(MapRecord.java:425)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:483)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$3.childNodeChanged(MapRecord.java:430)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:422)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:148)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord$2.childNodeChanged(MapRecord.java:389)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:473)
      	at org.apache.jackrabbit.oak.plugins.segment.MapRecord.compare(MapRecord.java:380)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:540)
      	at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:52)
      	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.updateIndex(AsyncIndexUpdate.java:354)
      	at org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate.run(AsyncIndexUpdate.java:300)
      	- locked <0x000000068afa00e8> (a org.apache.jackrabbit.oak.plugins.index.AsyncIndexUpdate)
      	at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:105)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:207)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
       "FelixStartLevel":
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentWriter.flush(SegmentWriter.java:167)
      	- waiting to lock <0x0000000680c634a8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentWriter)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.flush(FileStore.java:431)
      	- locked <0x000000068461ecb8> (a java.util.concurrent.atomic.AtomicReference)
      	at org.apache.jackrabbit.oak.plugins.segment.file.FileStore.close(FileStore.java:582)
      	- locked <0x0000000680024468> (a org.apache.jackrabbit.oak.plugins.segment.file.FileStore)
      	at org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService.deactivate(SegmentNodeStoreService.java:300)
      	- locked <0x00000006800243c8> (a org.apache.jackrabbit.oak.plugins.segment.SegmentNodeStoreService)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.felix.scr.impl.helper.BaseMethod.invokeMethod(BaseMethod.java:231)
      	at org.apache.felix.scr.impl.helper.BaseMethod.access$500(BaseMethod.java:39)
      	at org.apache.felix.scr.impl.helper.BaseMethod$Resolved.invoke(BaseMethod.java:624)
      	at org.apache.felix.scr.impl.helper.BaseMethod.invoke(BaseMethod.java:508)
      	at org.apache.felix.scr.impl.helper.ActivateMethod.invoke(ActivateMethod.java:149)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.disposeImplementationObject(SingleComponentManager.java:355)
      	at org.apache.felix.scr.impl.manager.SingleComponentManager.deleteComponent(SingleComponentManager.java:170)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.doDeactivate(AbstractComponentManager.java:908)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.deactivateInternal(AbstractComponentManager.java:883)
      	at org.apache.felix.scr.impl.manager.AbstractComponentManager.dispose(AbstractComponentManager.java:580)
      	at org.apache.felix.scr.impl.config.ConfigurableComponentHolder.disposeComponents(ConfigurableComponentHolder.java:406)
      	at org.apache.felix.scr.impl.BundleComponentActivator.dispose(BundleComponentActivator.java:335)
      	at org.apache.felix.scr.impl.Activator.disposeComponents(Activator.java:313)
      	at org.apache.felix.scr.impl.Activator.access$300(Activator.java:45)
      	at org.apache.felix.scr.impl.Activator$ScrExtension.destroy(Activator.java:198)
      	at org.apache.felix.utils.extender.AbstractExtender$2.run(AbstractExtender.java:290)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at org.apache.felix.utils.extender.AbstractExtender.destroyExtension(AbstractExtender.java:312)
      	at org.apache.felix.utils.extender.AbstractExtender.bundleChanged(AbstractExtender.java:186)
      	at org.apache.felix.framework.util.EventDispatcher.invokeBundleListenerCallback(EventDispatcher.java:869)
      	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:790)
      	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:515)
      	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4409)
      	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2526)
      	at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1315)
      	at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
      	at java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stillalex Alex Deparvu
                Reporter:
                stillalex Alex Deparvu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: