[ASTERIXDB-3400] Possible deadlock in GlobalVirtualBufferCache

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: IDX - Indexes
• Labels: triaged
• Assignee: Peeyush Gupta
• Reporter: Peeyush Gupta
• Created: 16/May/24 17:00
• Updated: 08/Jun/24 00:43
• Resolved: 08/Jun/24 00:43

Description

There is a possibility of deadlock while dropping and index and flush operation are happening simultaneously. Following show the deadlocked threads stack trace

"Executor-2784:622b93ec1479bc98eaffbc87b93ca050" [tid=3270 state=BLOCKED lock=org.apache.asterix.common.context.PrimaryIndexOperationTracker@2f1ba30a lockOwner="SAO:JID:0.807:TAID:TID:ANID:ODID:0:0:39:0" (tid=3314]	at app//org.apache.asterix.common.context.GlobalVirtualBufferCache.flushed(GlobalVirtualBufferCache.java:184)	- <locked org.apache.asterix.common.context.GlobalVirtualBufferCache@7af81de2>	at app//org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMMemoryComponent.flushed(AbstractLSMMemoryComponent.java:349)	at app//org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.doExitComponents(LSMHarness.java:284)	at app//org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.exitComponents(LSMHarness.java:360)	at app//org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.flush(LSMHarness.java:540)	at app//org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.flush(LSMTreeIndexAccessor.java:123)	at app//org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:38)	at app//org.apache.hyracks.storage.am.lsm.common.impls.FlushOperation.call(FlushOperation.java:29)	at java.base@17.0.10/java.util.concurrent.FutureTask.run(FutureTask.java:264)	at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)	at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)	at java.base@17.0.10/java.lang.Thread.run(Thread.java:840)

"SAO:JID:0.807:TAID:TID:ANID:ODID:0:0:39:0" [tid=3314 state=BLOCKED lock=org.apache.asterix.common.context.GlobalVirtualBufferCache@7af81de2 lockOwner="Executor-2784:622b93ec1479bc98eaffbc87b93ca050" (tid=3270]	at app//org.apache.asterix.common.context.GlobalVirtualBufferCache.unregister(GlobalVirtualBufferCache.java:143)	at app//org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMMemoryComponent.deallocate(AbstractLSMMemoryComponent.java:308)	at app//org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.deallocateMemoryComponents(AbstractLSMIndex.java:264)	at app//org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.deactivate(AbstractLSMIndex.java:244)	- <locked org.apache.hyracks.storage.am.lsm.btree.column.impls.lsm.LSMColumnBTree@6f64ebfe>	at app//org.apache.asterix.common.context.DatasetLifecycleManager.closeIndex(DatasetLifecycleManager.java:631)	- <locked org.apache.asterix.common.context.PrimaryIndexOperationTracker@2f1ba30a>	at app//org.apache.asterix.common.context.DatasetLifecycleManager.unregister(DatasetLifecycleManager.java:171)	- <locked org.apache.asterix.common.context.DatasetLifecycleManager@3e63567b>	at app//org.apache.hyracks.storage.am.common.dataflow.IndexDataflowHelper.destroy(IndexDataflowHelper.java:96)	- <locked org.apache.asterix.common.context.DatasetLifecycleManager@3e63567b>	at app//org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.dropIndex(IndexDropOperatorNodePushable.java:90)	at app//org.apache.hyracks.storage.am.common.dataflow.IndexDropOperatorNodePushable.initialize(IndexDropOperatorNodePushable.java:78)	at app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$1682/0x0000000800943f20.run(Unknown Source)	at app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$runInParallel$0(SuperActivityOperatorNodePushable.java:233)	at app//org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$$Lambda$1686/0x00000008009449c8.call(Unknown Source)	at java.base@17.0.10/java.util.concurrent.FutureTask.run(FutureTask.java:264)	at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)	at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)	at java.base@17.0.10/java.lang.Thread.run(Thread.java:840)