Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11331

[blockcache] lazy block decompression

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.99.0, 0.98.7
    • regionserver
    • None
    • Hide
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching.

      For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.
      Show
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching. For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.

    Description

      Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk.

      This is related to but less invasive than HBASE-8894.

      Attachments

        1. HBASE-11331.00.patch
          44 kB
          Nick Dimiduk
        2. HBASE-11331.01.patch
          52 kB
          Nick Dimiduk
        3. HBASE-11331.02.patch
          78 kB
          Nick Dimiduk
        4. HBASE-11331.03.patch
          101 kB
          Nick Dimiduk
        5. HBASE-11331.04.patch
          101 kB
          Nick Dimiduk
        6. HBASE-11331.05.patch
          101 kB
          Nick Dimiduk
        7. HBASE-11331.05-0.98.patch
          100 kB
          Nick Dimiduk
        8. HBASE-11331.06.patch
          110 kB
          Nick Dimiduk
        9. HBASE-11331.06-0.98.patch
          110 kB
          Nick Dimiduk
        10. HBASE-11331.07.patch
          110 kB
          Nick Dimiduk
        11. HBASE-11331.07-0.98.patch
          110 kB
          Nick Dimiduk
        12. HBASE-11331LazyBlockDecompressperfcompare.pdf
          354 kB
          Michael Stack
        13. hbase-hbase-master-hor17n36.gq1.ygridcore.net.log
          610 kB
          Nick Dimiduk
        14. lazy-decompress.02.0.pdf
          430 kB
          Nick Dimiduk
        15. lazy-decompress.02.1.json
          27 kB
          Nick Dimiduk
        16. lazy-decompress.02.1.pdf
          512 kB
          Nick Dimiduk
        17. v03-20g-045g-false.pdf
          189 kB
          Nick Dimiduk
        18. v03-20g-045g-true.pdf
          183 kB
          Nick Dimiduk
        19. v03-20g-045g-true-16h.pdf
          241 kB
          Nick Dimiduk

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. HBASE-12123 Failed assertion in BucketCache after HBASE-11331

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-11939 Document compressed blockcache

          • Major - Major loss of function.
          • Closed
          links to

          Web Link reviewboard

          Activity

            People

              ndimiduk Nick Dimiduk
              ndimiduk Nick Dimiduk
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: