Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml.
      Note that replication is currently broken when WAL compression is enabled.
      Show
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml. Note that replication is currently broken when WAL compression is enabled.

      Description

      The current bottleneck to HBase write speed is replicating the WAL appends across different datanodes. We can speed up this process by compressing the HLog. Current plan involves using a dictionary to compress table name, region id, cf name, and possibly other bits of repeated data. Also, HLog format may be changed in other ways to produce a smaller HLog.

        Attachments

        1. hbase-4608-v28-delta.txt
          25 kB
          Todd Lipcon
        2. hbase-4608-v28.txt
          56 kB
          Todd Lipcon
        3. hbase-4608-v28.txt
          56 kB
          stack
        4. 4608v8fixed.txt
          37 kB
          Li Pi
        5. 4608v7.txt
          32 kB
          Li Pi
        6. 4608v6.txt
          32 kB
          Li Pi
        7. 4608v5.txt
          33 kB
          Li Pi
        8. 4608v30.txt
          57 kB
          stack
        9. 4608v29.txt
          56 kB
          stack
        10. 4608v27.txt
          52 kB
          stack
        11. 4608v25.txt
          52 kB
          stack
        12. 4608v24.txt
          52 kB
          stack
        13. 4608v23.txt
          51 kB
          stack
        14. 4608-v22.txt
          42 kB
          Ted Yu
        15. 4608-v20.txt
          42 kB
          Ted Yu
        16. 4608-v19.txt
          41 kB
          Ted Yu
        17. 4608v18.txt
          39 kB
          Ted Yu
        18. 4608v17.txt
          39 kB
          Ted Yu
        19. 4608v16.txt
          39 kB
          Ted Yu
        20. 4608v15.txt
          39 kB
          Ted Yu
        21. 4608v14.txt
          40 kB
          Li Pi
        22. 4608v13.txt
          39 kB
          Li Pi
        23. 4608v13.txt
          39 kB
          Li Pi
        24. 4608v1.txt
          11 kB
          Li Pi

          Issue Links

            Activity

              People

              • Assignee:
                li Li Pi
                Reporter:
                li Li Pi
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: