HBase
  1. HBase
  2. HBASE-5778

Fix HLog compression's incompatibilities

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.94.4, 0.95.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I ran some tests to verify if WAL compression should be turned on by default.

      For a use case where it's not very useful (values two order of magnitude bigger than the keys), the insert time wasn't different and the CPU usage 15% higher (150% CPU usage VS 130% when not compressing the WAL).

      When values are smaller than the keys, I saw a 38% improvement for the insert run time and CPU usage was 33% higher (600% CPU usage VS 450%). I'm not sure WAL compression accounts for all the additional CPU usage, it might just be that we're able to insert faster and we spend more time in the MemStore per second (because our MemStores are bad when they contain tens of thousands of values).

      Those are two extremes, but it shows that for the price of some CPU we can save a lot. My machines have 2 quads with HT, so I still had a lot of idle CPUs.

      1. HBASE-5778.patch
        0.8 kB
        Jean-Daniel Cryans
      2. 5778.addendum
        1.0 kB
        Ted Yu
      3. 5778-addendum.txt
        3 kB
        Lars Hofhansl
      4. HBASE-5778-0.94.patch
        11 kB
        Jean-Daniel Cryans
      5. HBASE-5778-0.94-v2.patch
        13 kB
        Jean-Daniel Cryans
      6. HBASE-5778-0.94-v3.patch
        20 kB
        Jean-Daniel Cryans
      7. HBASE-5778-0.94-v4.patch
        19 kB
        Jean-Daniel Cryans
      8. HBASE-5778-0.94-v5.patch
        21 kB
        Jean-Daniel Cryans
      9. HBASE-5778-0.94-v6.patch
        21 kB
        Jean-Daniel Cryans
      10. HBASE-5778-trunk-v6.patch
        22 kB
        Jean-Daniel Cryans
      11. HBASE-5778-trunk-v7.patch
        25 kB
        Jean-Daniel Cryans
      12. HBASE-5778-0.94-v7.patch
        26 kB
        Jean-Daniel Cryans

        Issue Links

          Activity

          Jean-Daniel Cryans created issue -
          Jean-Daniel Cryans made changes -
          Field Original Value New Value
          Attachment HBASE-5778.patch [ 12522498 ]
          Jean-Daniel Cryans made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Release Note To disable WAL compression set hbase.regionserver.wal.enablecompression to false.
          Assignee Jean-Daniel Cryans [ jdcryans ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Assignee Jean-Daniel Cryans [ jdcryans ] Lars Hofhansl [ lhofhansl ]
          Ted Yu made changes -
          Attachment 5778.addendum [ 12522526 ]
          Lars Hofhansl made changes -
          Attachment 5778-addendum.txt [ 12522527 ]
          Lars Hofhansl made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Ted Yu made changes -
          Link This issue is related to HBASE-4608 [ HBASE-4608 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.1 [ 12320257 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.1 [ 12320257 ]
          Michael Drzal made changes -
          Link This issue relates to HBASE-6533 [ HBASE-6533 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94.patch [ 12552584 ]
          Jean-Daniel Cryans made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee Lars Hofhansl [ lhofhansl ] Jean-Daniel Cryans [ jdcryans ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v2.patch [ 12552694 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v3.patch [ 12553269 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v3.patch [ 12553269 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v3.patch [ 12553270 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v4.patch [ 12553808 ]
          stack made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          stack made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v5.patch [ 12560333 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-0.94-v6.patch [ 12561042 ]
          Attachment HBASE-5778-trunk-v6.patch [ 12561043 ]
          Jean-Daniel Cryans made changes -
          Attachment HBASE-5778-trunk-v7.patch [ 12561640 ]
          Attachment HBASE-5778-0.94-v7.patch [ 12561641 ]
          Ted Yu made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jean-Daniel Cryans made changes -
          Summary Turn on WAL compression by default Fix HLog compression's incompatibilities
          Release Note To disable WAL compression set hbase.regionserver.wal.enablecompression to false.
          Fix Version/s 0.94.4 [ 12323367 ]
          Jean-Daniel Cryans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Lars Hofhansl made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Lars Hofhansl made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          stack made changes -
          Fix Version/s 0.95.0 [ 12324094 ]
          Fix Version/s 0.96.0 [ 12320040 ]
          Fix Version/s 0.94.4 [ 12323367 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.0 [ 12316419 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.1 [ 12320257 ]
          Fix Version/s 0.94.0 [ 12316419 ]
          Lars Hofhansl made changes -
          Fix Version/s 0.94.4 [ 12323367 ]
          Fix Version/s 0.94.1 [ 12320257 ]

            People

            • Assignee:
              Jean-Daniel Cryans
              Reporter:
              Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development