[HBASE-20542] Better heap utilization for IMC with MSLABs

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0
• Component/s: in-memory-compaction
• Assignee: Eshcar Hillel
• Reporter: Eshcar Hillel
• Created: 08/May/18 11:08
• Updated: 28/Jan/19 13:38
• Resolved: 28/Jan/19 13:38

Description

Following HBASE-20188 we realized in-memory compaction combined with MSLABs may suffer from heap under-utilization due to internal fragmentation. This jira presents a solution to circumvent this problem. The main idea is to have each update operation check if it will cause overflow in the active segment before it is writing the new value (instead of checking the size after the write is completed), and if it is then the active segment is atomically swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to the compaction pipeline. Later on the IMC deamon will run its compaction operation (flatten index/merge indices/data compaction) in the background. Some subtle concurrency issues should be handled with care. We next elaborate on them.

Issue Links

• links to: RB link