HBase
  1. HBase
  2. HBASE-7404

Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.94.3
    • Fix Version/s: 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      BucketCache is another implementation of BlockCache which supports big block cache for high performance and would greatly decrease CMS and heap fragmentation in JVM caused by read activities.


      Usage:

      1.Use bucket cache as main memory cache, configured as the following:
      –"hbase.bucketcache.ioengine" "heap"
      –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

      2.Use bucket cache as a secondary cache, configured as the following:
      –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
      –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
      –"hbase.bucketcache.combinedcache.enabled" false (default value being true)
      Show
      BucketCache is another implementation of BlockCache which supports big block cache for high performance and would greatly decrease CMS and heap fragmentation in JVM caused by read activities. Usage: 1.Use bucket cache as main memory cache, configured as the following: –"hbase.bucketcache.ioengine" "heap" –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size) 2.Use bucket cache as a secondary cache, configured as the following: –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data) –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB) –"hbase.bucketcache.combinedcache.enabled" false (default value being true)
    • Tags:
      0.96notable

      Description

      First, thanks @neil from Fusion-IO share the source code.

      Usage:

      1.Use bucket cache as main memory cache, configured as the following:
      –"hbase.bucketcache.ioengine" "heap" (or "offheap" if using offheap memory to cache block )
      –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

      2.Use bucket cache as a secondary cache, configured as the following:
      –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
      –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
      –"hbase.bucketcache.combinedcache.enabled" false (default value being true)

      See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache

      What's Bucket Cache?
      It could greatly decrease CMS and heap fragment by GC
      It support a large cache space for High Read Performance by using high speed disk like Fusion-io

      1.An implementation of block cache like LruBlockCache
      2.Self manage blocks' storage position through Bucket Allocator
      3.The cached blocks could be stored in the memory or file system
      4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
      5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space

      How about SlabCache?
      We have studied and test SlabCache first, but the result is bad, because:
      1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
      2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
      3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

      See more in the attachment and in the patch

      1. BucketCache.pdf
        292 kB
        chunhui shen
      2. hbase-7404-trunkv2.patch
        125 kB
        chunhui shen
      3. hbase-7404-94v2.patch
        122 kB
        chunhui shen
      4. Introduction of Bucket Cache.pdf
        966 kB
        chunhui shen
      5. hbase-7404-trunkv9.patch
        130 kB
        chunhui shen
      6. 7404-trunk-v10.patch
        131 kB
        Ted Yu
      7. 7404-trunk-v11.patch
        131 kB
        Ted Yu
      8. 7404-trunk-v12.patch
        131 kB
        chunhui shen
      9. 7404-trunk-v13.patch
        133 kB
        chunhui shen
      10. 7404-trunk-v13.txt
        133 kB
        chunhui shen
      11. 7404-trunk-v14.patch
        133 kB
        chunhui shen
      12. HBASE-7404-backport-0.94.patch
        152 kB
        Dave Latham
      13. 7404-0.94-fixed-lines.txt
        131 kB
        Lars Hofhansl

        Issue Links

          Activity

          Jeff Hammerbacher made changes -
          Link This issue relates to HBASE-8894 [ HBASE-8894 ]
          chunhui shen made changes -
          Description First, thanks @neil from Fusion-IO share the source code.

          Usage:

          1.Use bucket cache as main memory cache, configured as the following:
          –"hbase.bucketcache.ioengine" "heap"
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configured as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false (default value being true)

          See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache


          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          Usage:

          1.Use bucket cache as main memory cache, configured as the following:
          –"hbase.bucketcache.ioengine" "heap" (or "offheap" if using offheap memory to cache block )
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configured as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false (default value being true)

          See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache


          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          stack made changes -
          Tags 0.96notable
          Lars Hofhansl made changes -
          Attachment 7404-0.94-fixed-lines.txt [ 12605769 ]
          stack made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Dave Latham made changes -
          Attachment HBASE-7404-backport-0.94.patch [ 12578433 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          stack made changes -
          Fix Version/s 0.98.0 [ 12323143 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.5 [ 12323874 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.5 [ 12323874 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          chunhui shen made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          chunhui shen made changes -
          Fix Version/s 0.94.5 [ 12323874 ]
          Ted Yu made changes -
          Release Note BucketCache is another implementation of BlockCache, which supports big block cache for high performance and could greatly decrease CMS and heap fragment caused by reading. BucketCache is another implementation of BlockCache which supports big block cache for high performance and would greatly decrease CMS and heap fragmentation in JVM caused by read activities.


          Usage:

          1.Use bucket cache as main memory cache, configured as the following:
          –"hbase.bucketcache.ioengine" "heap"
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configured as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false (default value being true)
          Description First, thanks @neil from Fusion-IO share the source code.

          Usage:

          1.Use bucket cache as a mainly memory cache, configure as the following:
          –"hbase.bucketcache.ioengine" "heap"
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configure as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false

          See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache


          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          Usage:

          1.Use bucket cache as main memory cache, configured as the following:
          –"hbase.bucketcache.ioengine" "heap"
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configured as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false (default value being true)

          See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache


          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          chunhui shen made changes -
          Release Note BucketCache is another implementation of BlockCache, which supports big block cache for high performance and could greatly decrease CMS and heap fragment caused by reading.
          chunhui shen made changes -
          Description First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          Usage:

          1.Use bucket cache as a mainly memory cache, configure as the following:
          –"hbase.bucketcache.ioengine" "heap"
          –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of max heap size)

          2.Use bucket cache as a secondary cache, configure as the following:
          –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path where to store the block data)
          –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024 means 1GB)
          –"hbase.bucketcache.combinedcache.enabled" false

          See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache


          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          chunhui shen made changes -
          Attachment 7404-trunk-v14.patch [ 12563398 ]
          chunhui shen made changes -
          Attachment 7404-trunk-v13.txt [ 12563244 ]
          chunhui shen made changes -
          Attachment 7404-trunk-v13.patch [ 12563050 ]
          chunhui shen made changes -
          Attachment 7404-trunk-v12.patch [ 12562905 ]
          chunhui shen made changes -
          Attachment 7404-trunk-v12.patch [ 12562904 ]
          chunhui shen made changes -
          Attachment 7404-trunk-v12.patch [ 12562904 ]
          Ted Yu made changes -
          Attachment 7404-trunk-v11.patch [ 12562848 ]
          Ted Yu made changes -
          Attachment 7404-trunk-v10.patch [ 12562846 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          chunhui shen made changes -
          Attachment hbase-7404-trunkv9.patch [ 12562833 ]
          chunhui shen made changes -
          Attachment Introduction of Bucket Cache.pdf [ 12562209 ]
          chunhui shen made changes -
          Attachment hbase-7404-trunkv1.patch [ 12562036 ]
          chunhui shen made changes -
          Attachment hbase-7404-94v1.patch [ 12562037 ]
          chunhui shen made changes -
          Attachment hbase-7404-trunkv2.patch [ 12562063 ]
          Attachment hbase-7404-94v2.patch [ 12562064 ]
          chunhui shen made changes -
          Attachment hbase-7404-trunkv1.patch [ 12562036 ]
          Attachment hbase-7404-94v1.patch [ 12562037 ]
          chunhui shen made changes -
          Attachment hbase-7404-trunkv1.patch [ 12561891 ]
          chunhui shen made changes -
          Attachment hbase-7404-0.94v1.patch [ 12561890 ]
          chunhui shen made changes -
          Description First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?

          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we doesn't do a test on "offheap" engine

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          We have studied and test SlabCache first, but the result is bad, because:
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using "heap" engine

          See more in the attachment and in the patch

          chunhui shen made changes -
          Description First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we doesn't do a test on "offheap" engine

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?

          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we doesn't do a test on "offheap" engine

          See more in the attachment and in the patch

          chunhui shen made changes -
          Description First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space

          See more in the attachment and in the patch

          First, thanks @neil from Fusion-IO share the source code.

          What's Bucket Cache?
          It could greatly decrease CMS and heap fragment by GC
          It support a large cache space for High Read Performance by using high speed disk like Fusion-io

          1.An implementation of block cache like LruBlockCache
          2.Self manage blocks' storage position through Bucket Allocator
          3.The cached blocks could be stored in the memory or file system
          4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC.
          5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space


          How about SlabCache?
          1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding
          2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better
          3.Direct heap performance is not good as heap, and maybe cause OOM, so we doesn't do a test on "offheap" engine

          See more in the attachment and in the patch

          Andrew Purtell made changes -
          Link This issue relates to HBASE-5487 [ HBASE-5487 ]
          Andrew Purtell made changes -
          Link This issue relates to HBASE-6572 [ HBASE-6572 ]
          chunhui shen made changes -
          Field Original Value New Value
          Attachment hbase-7404-0.94v1.patch [ 12561890 ]
          Attachment hbase-7404-trunkv1.patch [ 12561891 ]
          Attachment BucketCache.pdf [ 12561892 ]
          chunhui shen created issue -

            People

            • Assignee:
              chunhui shen
              Reporter:
              chunhui shen
            • Votes:
              7 Vote for this issue
              Watchers:
              52 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development