Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-14477

Compaction improvements: Date tiered compaction policy

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • None
    • 2.0.0
    • None
    • None

    Description

      For immutable and mostly immutable data the current SizeTiered-based compaction policy is not efficient.

      1. There is no need to compact all files into one, because, data is (mostly) immutable and we do not need to collect garbage. (performance reason will be discussed later)
      2. Size-tiered compaction is not suitable for applications where most recent data is most important and prevents efficient caching of this data.

      The idea is pretty similar to DateTieredCompaction in Cassandra:

      http://www.datastax.com/dev/blog/datetieredcompactionstrategy
      http://www.datastax.com/dev/blog/dtcs-notes-from-the-field

      From Cassandra own blog:

      Since DTCS can be used with any table, it is important to know when it is a good idea, and when it is not. I’ll try to explain the spectrum and trade-offs here:

      1. Perfect Fit: Time Series Fact Data, Deletes by Default TTL: When you ingest fact data that is ordered in time, with no deletes or overwrites. This is the standard “time series” use case.

      2. OK Fit: Time-Ordered, with limited updates across whole data set, or only updates to recent data: When you ingest data that is (mostly) ordered in time, but revise or delete a very small proportion of the overall data across the whole timeline.

      3. Not a Good Fit: many partial row updates or deletions over time: When you need to partially revise or delete fields for rows that you read together. Also, when you revise or delete rows within clustered reads.

      Attachments

        Issue Links

          duplicates

          New Feature - A new feature of the product, which has yet to be developed. HBASE-15181 A simple implementation of date based tiered compaction

          • Major - Major loss of function.
          • Closed
          is part of

          Improvement - An improvement or enhancement to an existing feature or task. HBASE-14383 Compaction improvements

          • Major - Major loss of function.
          • Closed
          relates to

          New Feature - A new feature of the product, which has yet to be developed. HBASE-9260 Timestamp Compactions

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. HBASE-7055 port HBASE-6371 tier-based compaction from 0.89-fb to trunk (with changes)

          • Major - Major loss of function.
          • Closed
          requires

          New Feature - A new feature of the product, which has yet to be developed. HBASE-14511 StoreFile.Writer Meta Plugin

          • Major - Major loss of function.
          • Closed

          Activity

            People

              Unassigned Unassigned
              vrodionov Vladimir Rodionov
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: