Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml.
      Note that replication is currently broken when WAL compression is enabled.
      Show
      Adds a custom dictionary-based compression on WAL. Off by default. To enable, set hbase.regionserver.wal.enablecompression to true in hbase-site.xml. Note that replication is currently broken when WAL compression is enabled.

      Description

      The current bottleneck to HBase write speed is replicating the WAL appends across different datanodes. We can speed up this process by compressing the HLog. Current plan involves using a dictionary to compress table name, region id, cf name, and possibly other bits of repeated data. Also, HLog format may be changed in other ways to produce a smaller HLog.

        Attachments

        1. 4608v1.txt
          11 kB
          Li Pi
        2. 4608v13.txt
          39 kB
          Li Pi
        3. 4608v13.txt
          39 kB
          Li Pi
        4. 4608v14.txt
          40 kB
          Li Pi
        5. 4608v15.txt
          39 kB
          Ted Yu
        6. 4608v16.txt
          39 kB
          Ted Yu
        7. 4608v17.txt
          39 kB
          Ted Yu
        8. 4608v18.txt
          39 kB
          Ted Yu
        9. 4608-v19.txt
          41 kB
          Ted Yu
        10. 4608-v20.txt
          42 kB
          Ted Yu
        11. 4608-v22.txt
          42 kB
          Ted Yu
        12. 4608v23.txt
          51 kB
          stack
        13. 4608v24.txt
          52 kB
          stack
        14. 4608v25.txt
          52 kB
          stack
        15. 4608v27.txt
          52 kB
          stack
        16. 4608v29.txt
          56 kB
          stack
        17. 4608v30.txt
          57 kB
          stack
        18. 4608v5.txt
          33 kB
          Li Pi
        19. 4608v6.txt
          32 kB
          Li Pi
        20. 4608v7.txt
          32 kB
          Li Pi
        21. 4608v8fixed.txt
          37 kB
          Li Pi
        22. hbase-4608-v28.txt
          56 kB
          stack
        23. hbase-4608-v28.txt
          56 kB
          Todd Lipcon
        24. hbase-4608-v28-delta.txt
          25 kB
          Todd Lipcon

          Issue Links

            Activity

              People

              • Assignee:
                li Li Pi
                Reporter:
                li Li Pi
              • Votes:
                0 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: