Uploaded image for project: 'Apache AsterixDB'
  1. Apache AsterixDB
  2. ASTERIXDB-2156

Encounter error during feed, in both inverted index and rtree

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None
    • Environment:
      4nc 8 partitions per node. Linux

      Description

      Hi, all
      When we try cloudberry recently, we encounter the following error when we use feed on 0.3 billions twitter records.

      We can reproduce the following errors

      SEVERE: Failed merge operation on {"class" : "LSMInvertedIndex", "dir" : "/kolla/asterixdb/asterix/node4/8/storage/storage/partition_31/twitter/ds_tweet_idx_text_idx", "memory" : 2, "disk" : 14}
      java.lang.Error: Maximum lock count exceeded
              at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.fullTryAcquireShared(ReentrantReadWriteLock.java:528)
              at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryAcquireShared(ReentrantReadWriteLock.java:488)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
              at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
              at org.apache.hyracks.storage.common.buffercache.CachedPage.acquireReadLatch(CachedPage.java:120)
              at org.apache.hyracks.storage.am.btree.impls.BTree.acquireLatch(BTree.java:542)
              at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:570)
              at org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:596)
              at org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:198)
              at org.apache.hyracks.storage.am.btree.impls.BTree.access$300(BTree.java:69)
              at org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:902)
              at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndexRangeSearchCursor.open(OnDiskInvertedIndexRangeSearchCursor.java:74)
              at org.apache.hyracks.storage.am.lsm.invertedindex.ondisk.OnDiskInvertedIndex$OnDiskInvertedIndexAccessor.rangeSearch(OnDiskInvertedIndex.java:463)
              at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexRangeSearchCursor.open(LSMInvertedIndexRangeSearchCursor.java:68)
              at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.search(LSMInvertedIndex.java:223)
              at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndex.doMerge(LSMInvertedIndex.java:353)
              at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:669)
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:573)
              at org.apache.hyracks.storage.am.lsm.invertedindex.impls.LSMInvertedIndexAccessor.merge(LSMInvertedIndexAccessor.java:124)
              at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
              at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      or,

      Nov 07, 2017 3:39:55 PM org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness merge
      SEVERE: Failed merge operation on {"class" : "LSMRTreeWithAntiMatterTuples", "dir" : "/kolla/asterixdb/asterix/node4/1/storage/storage/partition_24/twitter/ds_tweet_idx_location_idx", "memory" : 2, "disk" : 19}
      org.apache.hyracks.algebricks.common.exceptions.NotImplementedException: Value provider for type missing is not implemented
              at org.apache.asterix.dataflow.data.nontagged.valueproviders.PrimitiveValueProviderFactory$1.getValue(PrimitiveValueProviderFactory.java:60)
              at org.apache.hyracks.storage.am.rtree.frames.RTreeNSMFrame.calculateMBRImpl(RTreeNSMFrame.java:131)
              at org.apache.hyracks.storage.am.rtree.frames.RTreeNSMFrame.adjustMBR(RTreeNSMFrame.java:152)
              at org.apache.hyracks.storage.am.rtree.impls.RTree$RTreeBulkLoader.propagateBulk(RTree.java:1047)
              at org.apache.hyracks.storage.am.rtree.impls.RTree$RTreeBulkLoader.add(RTree.java:948)
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMIndexBulkLoader.add(LSMIndexBulkLoader.java:54)
              at org.apache.hyracks.storage.am.lsm.common.impls.ChainedLSMDiskComponentBulkLoader.add(ChainedLSMDiskComponentBulkLoader.java:54)
              at org.apache.hyracks.storage.am.lsm.rtree.impls.LSMRTreeWithAntiMatterTuples.doMerge(LSMRTreeWithAntiMatterTuples.java:195)
              at org.apache.hyracks.storage.am.lsm.common.impls.AbstractLSMIndex.merge(AbstractLSMIndex.java:667)
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMHarness.merge(LSMHarness.java:574)
              at org.apache.hyracks.storage.am.lsm.common.impls.LSMTreeIndexAccessor.merge(LSMTreeIndexAccessor.java:128)
              at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:45)
              at org.apache.hyracks.storage.am.lsm.common.impls.MergeOperation.call(MergeOperation.java:30)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lwhay Wenhai Li
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: