Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14463

Severe performance downgrade when parallel reading a single key from BucketCache

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.14, 1.1.2
    • Fix Version/s: 1.2.0, 1.3.0, 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We store feature data of online items in HBase, do machine learning on these features, and supply the outputs to our online search engine. In such scenario we will launch hundreds of yarn workers and each worker will read all features of one item(i.e. single rowkey in HBase), so there'll be heavy parallel reading on a single rowkey.

      We were using LruCache but start to try BucketCache recently to resolve gc issue, and just as titled we have observed severe performance downgrade. After some analytics we found the root cause is the lock in BucketCache#getBlock, as shown below

            try {
              lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
              // ...
              if (bucketEntry.equals(backingMap.get(key))) {
                // ...
                int len = bucketEntry.getLength();
                Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
                    bucketEntry.deserializerReference(this.deserialiserMap));
      

      Since ioEnging.read involves array copy, it's much more time-costed than the operation in LruCache. And since we're using synchronized in IdLock#getLockEntry, parallel read dropping on the same bucket would be executed in serial, which causes a really bad performance.

      To resolve the problem, we propose to use ReentranceReadWriteLock in BucketCache, and introduce a new class called IdReadWriteLock to implement it.

        Attachments

        1. HBASE-14463.branch-0.98.patch
          28 kB
          Yu Li
        2. 14463-branch-1-v12.txt
          17 kB
          Ted Yu
        3. HBASE-14463_v12.patch
          18 kB
          Yu Li
        4. pe_use_same_keys.patch
          6 kB
          Yu Li
        5. test-results.tar.gz
          220 kB
          Yu Li
        6. GC_with_WeakObjectPool.png
          124 kB
          Yu Li
        7. TestBucketCache_with_IdReadWriteLock-latest.png
          236 kB
          Yu Li
        8. TestBucketCache_with_IdLock-latest.png
          239 kB
          Yu Li
        9. HBASE-14463_v12.patch
          18 kB
          Yu Li
        10. HBASE-14463_v11.patch
          19 kB
          Yu Li
        11. TestBucketCache-new_with_IdReadWriteLock.png
          233 kB
          Yu Li
        12. TestBucketCache-new_with_IdLock.png
          237 kB
          Yu Li
        13. HBASE-14463_v5.patch
          17 kB
          Yu Li
        14. HBASE-14463_v4.patch
          16 kB
          Yu Li
        15. TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png
          233 kB
          Yu Li
        16. HBASE-14463_v3.patch
          16 kB
          Yu Li
        17. HBASE-14463_v2.patch
          15 kB
          Yu Li
        18. TestBucketCache_with_IdReadWriteLock.png
          227 kB
          Yu Li
        19. TestBucketCache_with_IdLock.png
          235 kB
          Yu Li
        20. HBASE-14463.patch
          16 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                carp84 Yu Li
                Reporter:
                carp84 Yu Li
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: