Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14918 In-Memory MemStore Flush and Compaction
  3. HBASE-16608

Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • None
    • None
    • Reviewed
    • Hide
      The index-compation and data-compaction variants of CompactingMemStore are introduced. In both types the active (mutable) segment is periodically flushed-in-memory and is added as immutable segment in the compaction pipeline. The CompactingMemStore of index-compaction type is merging all immutable segments of the compacting pipeline into one. The merging of N segments is explained below. The CompactingMemStore of data-compaction type is compacting all immutable segments of the compacting pipeline into one. After the merge/compaction the old segments in the compacting pipeline are replaced with one new.

      Before explaining the process of merging N old segments into new one, note that segment structure includes ordered index that allows traversing the cells data efficiently. The merge is copying the ordered indexes of the old segments into one ordered index of new segment. No data is copied, no cells are filtered. Alternatively, in the process of compacting N old segments into new one, both data and index are copied. The old cells are filtered, meaning upon compaction unused versions of the cells are not copied so the new segment has less data then all old ones.

      This issue introduces only the merging ability and simplifies the user intervention for switching between types. The previous CompactingMemStore structure was added by HBASE-16420 and HBASE-16421. The future refinements of the policy or merging/compacting will come in HBASE-16417.

      In order to create a table with CompactingMemStore as a MemStore one should use:
      create ‘<tablename>’, {NAME => ‘<cfname>’, IN_MEMORY_COMPACTION => true}
      IN_MEMORY_COMPACTION default is false, so table created as following will have the known DefaultMemStore as a MemStore.
      create ‘<tablename>’, {NAME => ‘<cfname>’}

      The default type of CompactingMemStore is index-compaction. In order to change it to data-compaction one should add to the hbase-site.xml
      <property>
          <name>hbase.hregion.compacting.memstore.type</name>
          <value>data-compaction</value>
        </property>

      in addition to creating the table as following
      create ‘<tablename>’, {NAME => ‘<cfname>’, IN_MEMORY_COMPACTION => true}
      Show
      The index-compation and data-compaction variants of CompactingMemStore are introduced. In both types the active (mutable) segment is periodically flushed-in-memory and is added as immutable segment in the compaction pipeline. The CompactingMemStore of index-compaction type is merging all immutable segments of the compacting pipeline into one. The merging of N segments is explained below. The CompactingMemStore of data-compaction type is compacting all immutable segments of the compacting pipeline into one. After the merge/compaction the old segments in the compacting pipeline are replaced with one new. Before explaining the process of merging N old segments into new one, note that segment structure includes ordered index that allows traversing the cells data efficiently. The merge is copying the ordered indexes of the old segments into one ordered index of new segment. No data is copied, no cells are filtered. Alternatively, in the process of compacting N old segments into new one, both data and index are copied. The old cells are filtered, meaning upon compaction unused versions of the cells are not copied so the new segment has less data then all old ones. This issue introduces only the merging ability and simplifies the user intervention for switching between types. The previous CompactingMemStore structure was added by HBASE-16420 and HBASE-16421 . The future refinements of the policy or merging/compacting will come in HBASE-16417 . In order to create a table with CompactingMemStore as a MemStore one should use: create ‘<tablename>’, {NAME => ‘<cfname>’, IN_MEMORY_COMPACTION => true} IN_MEMORY_COMPACTION default is false, so table created as following will have the known DefaultMemStore as a MemStore. create ‘<tablename>’, {NAME => ‘<cfname>’} The default type of CompactingMemStore is index-compaction. In order to change it to data-compaction one should add to the hbase-site.xml <property>     <name>hbase.hregion.compacting.memstore.type</name>     <value>data-compaction</value>   </property> in addition to creating the table as following create ‘<tablename>’, {NAME => ‘<cfname>’, IN_MEMORY_COMPACTION => true}

    Attachments

      1. HBASE-16417-V02.patch
        32 kB
        Anastasia Braginsky
      2. HBASE-16417-V04.patch
        32 kB
        Anastasia Braginsky
      3. HBASE-16417-V06.patch
        67 kB
        Anastasia Braginsky
      4. HBASE-16417-V07.patch
        68 kB
        Anastasia Braginsky
      5. HBASE-16417-V08.patch
        70 kB
        Anastasia Braginsky
      6. HBASE-16417-V10.patch
        70 kB
        Anastasia Braginsky
      7. HBASE-16608-Final.patch
        96 kB
        Anoop Sam John
      8. HBASE-16608-Final.patch
        95 kB
        Anoop Sam John
      9. HBASE-16608-V01.patch
        86 kB
        Anastasia Braginsky
      10. HBASE-16608-V03.patch
        86 kB
        Anastasia Braginsky
      11. HBASE-16608-V04.patch
        87 kB
        Anastasia Braginsky
      12. HBASE-16608-V08.patch
        96 kB
        Anastasia Braginsky
      13. HBASE-16608-V09.patch
        93 kB
        Anoop Sam John
      14. HBASE-16608-V09.patch
        59 kB
        Anoop Sam John

      Activity

        People

          anastas Anastasia Braginsky
          anastas Anastasia Braginsky
          Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved: