Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14463

Severe performance downgrade when parallel reading a single key from BucketCache

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.98.14, 1.1.2
    • 1.2.0, 1.3.0, 2.0.0
    • None
    • None
    • Reviewed

    Description

      We store feature data of online items in HBase, do machine learning on these features, and supply the outputs to our online search engine. In such scenario we will launch hundreds of yarn workers and each worker will read all features of one item(i.e. single rowkey in HBase), so there'll be heavy parallel reading on a single rowkey.

      We were using LruCache but start to try BucketCache recently to resolve gc issue, and just as titled we have observed severe performance downgrade. After some analytics we found the root cause is the lock in BucketCache#getBlock, as shown below

            try {
              lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
              // ...
              if (bucketEntry.equals(backingMap.get(key))) {
                // ...
                int len = bucketEntry.getLength();
                Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
                    bucketEntry.deserializerReference(this.deserialiserMap));
      

      Since ioEnging.read involves array copy, it's much more time-costed than the operation in LruCache. And since we're using synchronized in IdLock#getLockEntry, parallel read dropping on the same bucket would be executed in serial, which causes a really bad performance.

      To resolve the problem, we propose to use ReentranceReadWriteLock in BucketCache, and introduce a new class called IdReadWriteLock to implement it.

      Attachments

        1. HBASE-14463.branch-0.98.patch
          28 kB
          Yu Li
        2. 14463-branch-1-v12.txt
          17 kB
          Ted Yu
        3. HBASE-14463_v12.patch
          18 kB
          Yu Li
        4. pe_use_same_keys.patch
          6 kB
          Yu Li
        5. test-results.tar.gz
          220 kB
          Yu Li
        6. GC_with_WeakObjectPool.png
          124 kB
          Yu Li
        7. TestBucketCache_with_IdReadWriteLock-latest.png
          236 kB
          Yu Li
        8. TestBucketCache_with_IdLock-latest.png
          239 kB
          Yu Li
        9. HBASE-14463_v12.patch
          18 kB
          Yu Li
        10. HBASE-14463_v11.patch
          19 kB
          Yu Li
        11. TestBucketCache-new_with_IdReadWriteLock.png
          233 kB
          Yu Li
        12. TestBucketCache-new_with_IdLock.png
          237 kB
          Yu Li
        13. HBASE-14463_v5.patch
          17 kB
          Yu Li
        14. HBASE-14463_v4.patch
          16 kB
          Yu Li
        15. TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png
          233 kB
          Yu Li
        16. HBASE-14463_v3.patch
          16 kB
          Yu Li
        17. HBASE-14463_v2.patch
          15 kB
          Yu Li
        18. TestBucketCache_with_IdReadWriteLock.png
          227 kB
          Yu Li
        19. TestBucketCache_with_IdLock.png
          235 kB
          Yu Li
        20. HBASE-14463.patch
          16 kB
          Yu Li

        Issue Links

          Activity

            People

              liyu Yu Li
              liyu Yu Li
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: