[ASTERIXDB-2183] LSMHarness merge fails

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: IDX - Indexes
• Labels: triaged
• Assignee: Ian Maxon
• Reporter: Taewoo Kim
• Created: 01/Dec/17 03:06
• Updated: 15/Mar/18 00:02

Description

LSMHarness merge() operation fails on indexes with the following error. All index types are affected.

Nov 30, 2017 6:58:07 PM org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness merge
SEVERE: Failed merge operation on

{"class" : "LSMInvertedIndex", "dir" : "/mnt/ssd/scratch/waans11/asterixdb/io1/storage/partition_4/twitter/ds_tweet_idx_text_idx", "memory" : 2, "disk" : 894}

java.lang.Error: Maximum lock count exceeded
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at org.apache.hyracks.storage.common.buffercache.CachedPage.acquireReadLatch(CachedPage.java:120)
at org.apache.hyracks.storage.am.btree.impls.BTree.acquireLatch(BTree.java:542)
at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:570)
at org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:198)
at org.apache.hyracks.storage.am.btree.impls.BTree.access$300(BTree.java:69)
at org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:902)
at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndexRangeSearchCursor.open(OnDiskInvertedIndexRangeSearchCursor.java:74)
at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndex$OnDiskInvertedIndexAccessor.rangeSearch(OnDiskInvertedIndex.java:463)
at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexRangeSearchCursor.open(LSMInvertedIndexRangeSearchCursor.java:68)
at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.search(LSMInvertedIndex.java:223)
at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.doMerge(LSMInvertedIndex.java:353)
at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:671)
at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:574)
at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexAccessor.merge(LSMInvertedIndexAccessor.java:124)
at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)

This error was generated continuously until the node becomes inactive.