1. HBase
  2. HBASE-5778

Fix HLog compression's incompatibilities


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:


      I ran some tests to verify if WAL compression should be turned on by default.

      For a use case where it's not very useful (values two order of magnitude bigger than the keys), the insert time wasn't different and the CPU usage 15% higher (150% CPU usage VS 130% when not compressing the WAL).

      When values are smaller than the keys, I saw a 38% improvement for the insert run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure WAL compression accounts for all the additional CPU usage, it might just be that we're able to insert faster and we spend more time in the MemStore per second (because our MemStores are bad when they contain tens of thousands of values).

      Those are two extremes, but it shows that for the price of some CPU we can save a lot. My machines have 2 quads with HT, so I still had a lot of idle CPUs.

      1. HBASE-5778-trunk-v7.patch
        25 kB
        Jean-Daniel Cryans
      2. HBASE-5778-trunk-v6.patch
        22 kB
        Jean-Daniel Cryans
      3. HBASE-5778-0.94-v7.patch
        26 kB
        Jean-Daniel Cryans
      4. HBASE-5778-0.94-v6.patch
        21 kB
        Jean-Daniel Cryans
      5. HBASE-5778-0.94-v5.patch
        21 kB
        Jean-Daniel Cryans
      6. HBASE-5778-0.94-v4.patch
        19 kB
        Jean-Daniel Cryans
      7. HBASE-5778-0.94-v3.patch
        20 kB
        Jean-Daniel Cryans
      8. HBASE-5778-0.94-v2.patch
        13 kB
        Jean-Daniel Cryans
      9. HBASE-5778-0.94.patch
        11 kB
        Jean-Daniel Cryans
      10. HBASE-5778.patch
        0.8 kB
        Jean-Daniel Cryans
      11. 5778-addendum.txt
        3 kB
        Lars Hofhansl
      12. 5778.addendum
        1.0 kB
        Ted Yu

        Issue Links



            • Assignee:
              Jean-Daniel Cryans
              Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              16 Start watching this issue


              • Created: