Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-4608

HLog Compression

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.94.0
    • None
    • None
    • Reviewed
    • Hide
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml.
      Note that replication is currently broken when WAL compression is enabled.
      Show
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml. Note that replication is currently broken when WAL compression is enabled.

    Description

      The current bottleneck to HBase write speed is replicating the WAL appends across different datanodes. We can speed up this process by compressing the HLog. Current plan involves using a dictionary to compress table name, region id, cf name, and possibly other bits of repeated data. Also, HLog format may be changed in other ways to produce a smaller HLog.

      Attachments

        1. 4608v1.txt
          11 kB
          Li Pi
        2. 4608v13.txt
          39 kB
          Li Pi
        3. 4608v13.txt
          39 kB
          Li Pi
        4. 4608v14.txt
          40 kB
          Li Pi
        5. 4608v15.txt
          39 kB
          Ted Yu
        6. 4608v16.txt
          39 kB
          Ted Yu
        7. 4608v17.txt
          39 kB
          Ted Yu
        8. 4608v18.txt
          39 kB
          Ted Yu
        9. 4608-v19.txt
          41 kB
          Ted Yu
        10. 4608-v20.txt
          42 kB
          Ted Yu
        11. 4608-v22.txt
          42 kB
          Ted Yu
        12. 4608v23.txt
          51 kB
          Michael Stack
        13. 4608v24.txt
          52 kB
          Michael Stack
        14. 4608v25.txt
          52 kB
          Michael Stack
        15. 4608v27.txt
          52 kB
          Michael Stack
        16. 4608v29.txt
          56 kB
          Michael Stack
        17. 4608v30.txt
          57 kB
          Michael Stack
        18. 4608v5.txt
          33 kB
          Li Pi
        19. 4608v6.txt
          32 kB
          Li Pi
        20. 4608v7.txt
          32 kB
          Li Pi
        21. 4608v8fixed.txt
          37 kB
          Li Pi
        22. hbase-4608-v28.txt
          56 kB
          Michael Stack
        23. hbase-4608-v28.txt
          56 kB
          Todd Lipcon
        24. hbase-4608-v28-delta.txt
          25 kB
          Todd Lipcon

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HBASE-8155 Allow Record Compression for HLogs

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-1783 Ability for HDFS client to write replicas in parallel

          • Major - Major loss of function.
          • Patch Available

          Task - A task that needs to be done. ACCUMULO-2632 Research the performance of using compression on the walogs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-5778 Fix HLog compression's incompatibilities

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Activity

            People

              li Li Pi
              li Li Pi
              Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: