Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-11331

[blockcache] lazy block decompression

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0, 0.98.7
    • Component/s: regionserver
    • Labels:
      None
    • Release Note:
      Hide
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching.

      For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.
      Show
      When hbase.block.data.cachecompressed=true, DATA (and ENCODED_DATA) blocks are cached in the BlockCache in their on-disk format. This is different from the default behavior, which decompresses and decrypts a block before caching. For a region server hosting more data than fits into cache, enabling this feature with SNAPPY compression results in 50% increase in throughput and 30% improvement in mean latency while increasing GC by 80% and increasing overall CPU load by 2%.

      Description

      Maintaining data in its compressed form in the block cache will greatly increase our effective blockcache size and should show a meaning improvement in cache hit rates in well designed applications. The idea here is to lazily decompress/decrypt blocks when they're consumed, rather than as soon as they're pulled off of disk.

      This is related to but less invasive than HBASE-8894.

        Attachments

        1. HBASE-11331.00.patch
          44 kB
          Nick Dimiduk
        2. HBASE-11331LazyBlockDecompressperfcompare.pdf
          354 kB
          stack
        3. HBASE-11331.01.patch
          52 kB
          Nick Dimiduk
        4. HBASE-11331.02.patch
          78 kB
          Nick Dimiduk
        5. lazy-decompress.02.0.pdf
          430 kB
          Nick Dimiduk
        6. lazy-decompress.02.1.pdf
          512 kB
          Nick Dimiduk
        7. lazy-decompress.02.1.json
          27 kB
          Nick Dimiduk
        8. HBASE-11331.03.patch
          101 kB
          Nick Dimiduk
        9. v03-20g-045g-false.pdf
          189 kB
          Nick Dimiduk
        10. v03-20g-045g-true.pdf
          183 kB
          Nick Dimiduk
        11. v03-20g-045g-true-16h.pdf
          241 kB
          Nick Dimiduk
        12. HBASE-11331.04.patch
          101 kB
          Nick Dimiduk
        13. HBASE-11331.05.patch
          101 kB
          Nick Dimiduk
        14. hbase-hbase-master-hor17n36.gq1.ygridcore.net.log
          610 kB
          Nick Dimiduk
        15. HBASE-11331.05-0.98.patch
          100 kB
          Nick Dimiduk
        16. HBASE-11331.06-0.98.patch
          110 kB
          Nick Dimiduk
        17. HBASE-11331.06.patch
          110 kB
          Nick Dimiduk
        18. HBASE-11331.07-0.98.patch
          110 kB
          Nick Dimiduk
        19. HBASE-11331.07.patch
          110 kB
          Nick Dimiduk

          Issue Links

            Activity

              People

              • Assignee:
                ndimiduk Nick Dimiduk
                Reporter:
                ndimiduk Nick Dimiduk
              • Votes:
                0 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: