Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-5778

Fix HLog compression's incompatibilities

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I ran some tests to verify if WAL compression should be turned on by default.

      For a use case where it's not very useful (values two order of magnitude bigger than the keys), the insert time wasn't different and the CPU usage 15% higher (150% CPU usage VS 130% when not compressing the WAL).

      When values are smaller than the keys, I saw a 38% improvement for the insert run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure WAL compression accounts for all the additional CPU usage, it might just be that we're able to insert faster and we spend more time in the MemStore per second (because our MemStores are bad when they contain tens of thousands of values).

      Those are two extremes, but it shows that for the price of some CPU we can save a lot. My machines have 2 quads with HT, so I still had a lot of idle CPUs.

        Attachments

        1. HBASE-5778.patch
          0.8 kB
          Jean-Daniel Cryans
        2. 5778.addendum
          1.0 kB
          Ted Yu
        3. 5778-addendum.txt
          3 kB
          Lars Hofhansl
        4. HBASE-5778-0.94.patch
          11 kB
          Jean-Daniel Cryans
        5. HBASE-5778-0.94-v2.patch
          13 kB
          Jean-Daniel Cryans
        6. HBASE-5778-0.94-v3.patch
          20 kB
          Jean-Daniel Cryans
        7. HBASE-5778-0.94-v4.patch
          19 kB
          Jean-Daniel Cryans
        8. HBASE-5778-0.94-v5.patch
          21 kB
          Jean-Daniel Cryans
        9. HBASE-5778-0.94-v6.patch
          21 kB
          Jean-Daniel Cryans
        10. HBASE-5778-trunk-v6.patch
          22 kB
          Jean-Daniel Cryans
        11. HBASE-5778-trunk-v7.patch
          25 kB
          Jean-Daniel Cryans
        12. HBASE-5778-0.94-v7.patch
          26 kB
          Jean-Daniel Cryans

          Issue Links

            Activity

              People

              • Assignee:
                jdcryans Jean-Daniel Cryans
                Reporter:
                jdcryans Jean-Daniel Cryans
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: