Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-17747

Support both weak and soft object pool

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      During YCSB testing on embedded mode after HBASE-17744, we found that under high read load GC is quite severe even with offheap L2 cache. After some investigation, we found it's caused by using weak reference in IdReadWriteLock. In embedded mode the read is so quick that the lock might already get promoted to the old generation when the weak reference is cleared, which causes dirty card table (old reference get removed and new lock object set into referenceCache, see WeakObjectPool#get) thus slowing YGC. In distributed mode there'll also be more lock object created with weak reference than soft reference that slowing down the processing.

      So we proposed to use soft reference for this IdReadWriteLock used in cache, which won't get cleared until JVM memory is not enough, and could resolve the issue mentioned above. What's more, we propose to extend the WeakObjectPool to be more generate to support both weak and soft reference.

      Note that the GC issue only emerges under embedded mode with DirectOperator, in which case all costs on the wire is removed thus produces extremely high concurrency.

        Attachments

        1. HBASE-17747.patch
          20 kB
          Yu Li
        2. HBASE-17747.v2.patch
          20 kB
          Yu Li
        3. HBASE-17747.v3.patch
          20 kB
          Yu Li

          Issue Links

            Activity

              People

              • Assignee:
                liyu Yu Li
                Reporter:
                liyu Yu Li
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: