Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14918

In-Memory MemStore Flush and Compaction

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Umbrella
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      A memstore serves as the in-memory component of a store unit, absorbing all updates to the store. From time to time these updates are flushed to a file on disk, where they are compacted (by eliminating redundancies) and compressed (i.e., written in a compressed format to reduce their storage size).

      We aim to speed up data access, and therefore suggest to apply in-memory memstore flush. That is to flush the active in-memory segment into an intermediate buffer where it can be accessed by the application. Data in the buffer is subject to compaction and can be stored in any format that allows it to take up smaller space in RAM. The less space the buffer consumes the longer it can reside in memory before data is flushed to disk, resulting in better performance.
      Specifically, the optimization is beneficial for workloads with medium-to-high key churn which incur many redundant cells, like persistent messaging.

      We suggest to structure the solution as 4 subtasks (respectively, patches).
      (1) Infrastructure - refactoring of the MemStore hierarchy, introducing segment (StoreSegment) as first-class citizen, and decoupling memstore scanner from the memstore implementation;
      (2) Adding StoreServices facility at the region level to allow memstores update region counters and access region level synchronization mechanism;
      (3) Implementation of a new memstore (CompactingMemstore) with non-optimized immutable segment representation, and
      (4) Memory optimization including compressed format representation and off heap allocations.

      This Jira continues the discussion in HBASE-13408.
      Design documents, evaluation results and previous patches can be found in HBASE-13408.

        Attachments

        1. CellBlocksSegmentDesign.pdf
          133 kB
          Eshcar Hillel
        2. HBASE-16417-benchmarkresults.pdf
          397 kB
          Eshcar Hillel
        3. MSLABMove.patch
          30 kB
          Anoop Sam John

        Issue Links

        1.
        Infrastructure refactoring for In-Memory Flush Sub-task Resolved Eshcar Hillel Actions
        2.
        StoreServices facility in Region Sub-task Resolved Eshcar Hillel Actions
        3.
        Simplifying Segment hierarchy Sub-task Resolved Eshcar Hillel Actions
        4.
        Compacting Memstore Sub-task Resolved Eshcar Hillel Actions
        5.
        Inmemory Compaction Optimizations; Segment Structure Sub-task Resolved Anastasia Braginsky Actions
        6.
        Purge rollback support in Store etc. Sub-task Resolved Michael Stack Actions
        7.
        In-Memory MemStore Policy for Flattening and Compactions Sub-task Resolved Eshcar Hillel Actions
        8.
        Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage Sub-task Resolved Anastasia Braginsky Actions
        9.
        Document for HBASE-14918 (in-memory flushes/compactions) Sub-task Resolved Unassigned Actions
        10.
        Run large scale correctness tests for HBASE-14918 (in-memory flushes/compactions) Sub-task Open Devaraj Das Actions
        11.
        User-facing documentation for the In-Memory Compaction feature Sub-task Resolved Edward Bortnikov Actions
        12.
        Flush the entire CompactingMemStore content to disk Sub-task Resolved Anastasia Braginsky Actions
        13.
        External Configuration for Memory Compaction Sub-task Resolved Eshcar Hillel Actions
        14.
        Addendum to HBASE-17294, 'External Configuration for Memory Compaction' Sub-task Resolved Eshcar Hillel Actions
        15.
        Reverse the order of snapshot creation in the CompactingMemStore Sub-task Resolved Eshcar Hillel Actions
        16.
        Fix the compacting memstore part in hbase shell ruby script Sub-task Resolved Anastasia Braginsky Actions
        17.
        Run critical tests with each of the Inmemory Compaction Policies enabled (Towards Making BASIC the Default In-Memory Compaction Policy) Sub-task Resolved Eshcar Hillel Actions
        18.
        Disable in-memory flush when replaying from WAL Sub-task Resolved Anastasia Braginsky Actions
        19.
        Reviving the merge possibility in the CompactingMemStore Sub-task Resolved Anastasia Braginsky Actions
        20.
        Change CompactingMemStore in BASIC mode to merge multiple segments in pipeline Sub-task Resolved Anastasia Braginsky Actions

          Activity

            People

            • Assignee:
              eshcar Eshcar Hillel
              Reporter:
              eshcar Eshcar Hillel

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment