[ASTERIXDB-2156] Encounter error during feed, in both inverted index and rtree

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: IDX - Indexes, ING - Ingestion
• Environment: 4nc 8 partitions per node. Linux
• Assignee: Unassigned
• Reporter: Wenhai Li
• Created: 07/Nov/17 07:43
• Updated: 15/Mar/18 00:16
• Resolved: 15/Mar/18 00:16

Description

Hi, all
When we try cloudberry recently, we encounter the following error when we use feed on 0.3 billions twitter records.

We can reproduce the following errors

SEVERE: Failed merge operation on {"class" : "LSMInvertedIndex", "dir" : "/kolla/asterixdb/asterix/node4/8/storage/storage/partition_31/twitter/ds_tweet_idx_text_idx", "memory" : 2, "disk" : 14}
java.lang.Error: Maximum lock count exceeded
        at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
        at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at org.apache.hyracks.storage.common.buffercache.CachedPage.acquireReadLatch(CachedPage.java:120)
        at org.apache.hyracks.storage.am.btree.impls.BTree.acquireLatch(BTree.java:542)
        at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:570)
        at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:596)
        at org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:198)
        at org.apache.hyracks.storage.am.btree.impls.BTree.access$300(BTree.java:69)
        at org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:902)
        at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndexRangeSearchCursor.open(OnDiskInvertedIndexRangeSearchCursor.java:74)
        at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndex$OnDiskInvertedIndexAccessor.rangeSearch(OnDiskInvertedIndex.java:463)
        at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexRangeSearchCursor.open(LSMInvertedIndexRangeSearchCursor.java:68)
        at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.search(LSMInvertedIndex.java:223)
        at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.doMerge(LSMInvertedIndex.java:353)
        at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:669)
        at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:573)
        at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexAccessor.merge(LSMInvertedIndexAccessor.java:124)
        at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
        at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

or,

Nov 07, 2017 3:39:55 PM org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness merge
SEVERE: Failed merge operation on {"class" : "LSMRTreeWithAntiMatterTuples", "dir" : "/kolla/asterixdb/asterix/node4/1/storage/storage/partition_24/twitter/ds_tweet_idx_location_idx", "memory" : 2, "disk" : 19}
org.apache.hyracks.algebricks.common.exceptions.NotImplementedException: Value provider for type missing is not implemented
        at org.apache.asterix.dataflow.data.nontagged.valueproviders.PrimitiveValueProviderFactory$1.getValue(PrimitiveValueProviderFactory.java:60)
        at org.apache.hyracks.storage.am.rtree.frames.RTreeNSMFrame.calculateMBRImpl(RTreeNSMFrame.java:131)
        at org.apache.hyracks.storage.am.rtree.frames.RTreeNSMFrame.adjustMBR(RTreeNSMFrame.java:152)
        at org.apache.hyracks.storage.am.rtree.impls.RTree$RTreeBulkLoader.propagateBulk(RTree.java:1047)
        at org.apache.hyracks.storage.am.rtree.impls.RTree$RTreeBulkLoader.add(RTree.java:948)
        at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexBulkLoader.add(LSMIndexBulkLoader.java:54)
        at org.apache.hyracks.storage.am.lsm.common.impls.ChainedLSMDiskComponentBulkLoader.add(ChainedLSMDiskComponentBulkLoader.java:54)
        at org.apache.hyracks.storage.am.lsm.rtree.impls.LSMRTreeWithAntiMatterTuples.doMerge(LSMRTreeWithAntiMatterTuples.java:195)
        at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:667)
        at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:574)
        at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:128)
        at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
        at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)