[ASTERIXDB-1171] Deadlock between DatasetLifeCycleManager and PrimaryIndexOperationTracker

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Component/s: STO - Storage
• Assignee: Murtadha Makki Al Hubail
• Reporter: Yingyi Bu
• Created: 12/Nov/15 17:36
• Updated: 12/Nov/15 21:02
• Resolved: 12/Nov/15 21:02

Description

This deadlock looks different from ASTERIXDB-1170, because it's not in NC shutdown.

I run into this issue when running ExecutionTest (on cross-dataverse/insert_across_dataverses ). JVM is able to report this deadlock:

Java stack information for the threads listed above:
===================================================
"org.apache.hyracks.api.rewriter.runtime.SuperActivity:TAID:TID:ANID:ODID:4:0:1:0:0":
at org.apache.asterix.common.context.DatasetLifecycleManager.open(DatasetLifecycleManager.java:205)

• waiting to lock <0x000000079f601228> (a org.apache.asterix.common.context.PrimaryIndexOperationTracker)
• locked <0x0000000753633080> (a org.apache.asterix.common.context.DatasetLifecycleManager)
  at org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.open(IndexDataflowHelper.java:116)
• locked <0x0000000753633080> (a org.apache.asterix.common.context.DatasetLifecycleManager)
  at org.apache.asterix.common.dataflow.AsterixLSMInsertDeleteOperatorNodePushable.open(AsterixLSMInsertDeleteOperatorNodePushable.java:61)
  at org.apache.hyracks.dataflow.std.misc.MaterializerTaskState.writeOut(MaterializerTaskState.java:71)
  at org.apache.hyracks.dataflow.std.misc.MaterializingOperatorDescriptor$MaterializerReaderActivityNode$1.close(MaterializingOperatorDescriptor.java:117)
  at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
  at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
  at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.close(AlgebricksMetaOperatorDescriptor.java:150)
  at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:202)
  at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
  at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:113)
  at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:85)
  at org.apache.hyracks.control.nc.Task.run(Task.java:255)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  "org.apache.hyracks.api.rewriter.runtime.SuperActivity:TAID:TID:ANID:ODID:4:0:0:0:0":
  at org.apache.asterix.common.context.DatasetLifecycleManager.allocateDatasetMemory(DatasetLifecycleManager.java:639)
• waiting to lock <0x0000000753633080> (a org.apache.asterix.common.context.DatasetLifecycleManager)
  at org.apache.asterix.common.context.PrimaryIndexOperationTracker.beforeOperation(PrimaryIndexOperationTracker.java:64)
  at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.enterComponents(LSMHarness.java:180)
  at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.getAndEnterComponents(LSMHarness.java:115)
• locked <0x000000079f601228> (a org.apache.asterix.common.context.PrimaryIndexOperationTracker)
  at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:336)
  at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.modify(LSMHarness.java:327)
  at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.insert(LSMTreeIndexAccessor.java:50)
  at org.apache.asterix.common.dataflow.AsterixLSMInsertDeleteOperatorNodePushable.nextFrame(AsterixLSMInsertDeleteOperatorNodePushable.java:102)
  at org.apache.hyracks.dataflow.std.misc.MaterializerTaskState.writeOut(MaterializerTaskState.java:75)
  at org.apache.hyracks.dataflow.std.misc.MaterializingOperatorDescriptor$MaterializerReaderActivityNode$1.close(MaterializingOperatorDescriptor.java:117)
  at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
  at org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
  at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$2.close(AlgebricksMetaOperatorDescriptor.java:150)
  at org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:202)
  at org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
  at org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:113)
  at org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:85)
  at org.apache.hyracks.control.nc.Task.run(Task.java:255)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.

Issue Links

• is part of: ASTERIXDB-1170 Deadlock in shutdown with DatasetLifecycleManager