HBase
  1. HBase
  2. HBASE-8163

MemStoreChunkPool: An improvement for JAVA GC when using MSLAB

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Support reusing MemStoreChunk when MSLAB is enabled, thereby decrease the cost of YGC. It is off by default, turn it on by configuring "hbase.hregion.memstore.chunkpool.maxsize" (between 0.0 and 1.0)

      Description

      Usage:
      Disable pool(default): configure "hbase.hregion.memstore.chunkpool.maxsize" as 0
      Enable pool: configure "hbase.hregion.memstore.chunkpool.maxsize" as a percentage of global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and max sizes of memstore, e.g. 0.5)

      Background
      When we use mslab,we will copy the keyvalue together in a structure called MemStoreLAB$Chunk, therefore we could decrease the heap fragment.

      Problem
      When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed by JVM if no reference to it.

      Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost of YGC

      When a Chunk object has no reference? It should meet two following condition:
      1.Memstore which this chunk belongs to is flushed
      2.No scanner is opening on the memstore which this chunk belongs to

      Solution:
      1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by JVM
      2.When a Chunk has no reference, put it back to the pool
      3.The pool has a max capacity, it will skip the chunks when achieve the max size
      4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create new one by pool, so we could be able to reuse the old chunks

      Test results:
      Environment:
      hbase-version:0.94
      -Xms4G -Xmx4G -Xmn2G
      Row size=50 bytes, Value size=1024 bytes
      50 concurrent theads per client, insert 10,000,000 rows

      Before:
      Avg write request per second:12953
      After testing, final result of jstat -gcutil :
      YGC YGCT FGC FGCT GCT
      747 36.503 48 2.492 38.995

      After:
      Avg write request per second:14025
      After testing, final result of jstat -gcutil :
      YGC YGCT FGC FGCT GCT
      711 20.344 4 0.284 20.628

      Improvement: YGC 40+%; WPS 5%+

      review board :
      https://reviews.apache.org/r/10056/

      1. hbase-0.95-8163v6.patch
        26 kB
        chunhui shen
      2. hbase-8163v6.patch
        26 kB
        chunhui shen
      3. hbase-8163v5.patch
        26 kB
        chunhui shen
      4. hbase-8163v4.patch
        26 kB
        chunhui shen
      5. hbase-8163v3.patch
        22 kB
        chunhui shen
      6. hbase-8163v2.patch
        22 kB
        chunhui shen
      7. hbase-8163v1.patch
        21 kB
        chunhui shen

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            chunhui shen
            Reporter:
            chunhui shen
          • Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development