Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0, 2.2.0
    • Component/s: in-memory-compaction
    • Labels:
      None

      Description

      Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment before it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them.

        Attachments

        1. HBASE-20542-addendum.master.005.patch
          7 kB
          Eshcar Hillel
        2. HBASE-20542.master.005-addendum.patch
          7 kB
          Eshcar Hillel
        3. HBASE-20542.branch-2.005.patch
          70 kB
          Eshcar Hillel
        4. HBASE-20542.master.003.patch
          70 kB
          Eshcar Hillel
        5. HBASE-20542.branch-2.004.patch
          69 kB
          Eshcar Hillel
        6. HBASE-20542.branch-2.003.patch
          68 kB
          Eshcar Hillel
        7. workloada
          3 kB
          Eshcar Hillel
        8. workloadc
          3 kB
          Eshcar Hillel
        9. run.sh
          3 kB
          Eshcar Hillel
        10. workloadx
          3 kB
          Eshcar Hillel
        11. workloady
          3 kB
          Eshcar Hillel
        12. HBASE-20542.branch-2.001.patch
          65 kB
          Eshcar Hillel

          Issue Links

            Activity

              People

              • Assignee:
                eshcar Eshcar Hillel
                Reporter:
                eshcar Eshcar Hillel
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: