This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #852 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/852/)
HBASE-9854 initial documentation for stripe compactions (stack: rev 1545382)

• /hbase/trunk/src/main/docbkx/book.xml

FAILURE: Integrated in HBase-TRUNK #4698 (See https://builds.apache.org/job/HBase-TRUNK/4698/)
HBASE-9854 initial documentation for stripe compactions (stack: rev 1545382)

• /hbase/trunk/src/main/docbkx/book.xml

Thanks!

Committed for Sergey Shelukhin

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12613960/HBASE-9854.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 lineLengths. The patch does not introduce lines longer than 100

-1 site. The patch appears to cause mvn site goal to fail.

+1 core tests. The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//console

This message is automatically generated.

Patch. I am not able to build mvn site, fails with OOM seemingly no matter what I do. But, it did catch syntax problems when there were some before going to OOM, so I assume it's mostly validated.

How about http://hbase.apache.org/book.html#compaction ?

Hmm... which part of doc should this go to?

I will make a doc patch, hopefully tomorrow or day after.
You do want to use it for uniform tables if your regions are large, at the cost of slightly more I/O (w/flush-into-stripes) you can get more stable behavior. In my tests on multi-Gb regions reads were also slightly faster with 10-12 stripes, although I didn't investigate why then

Sergey Shelukhin Very nice doc especially the way you introduce the feature. Want me to add it into the doc for you?

To set it back to default, should you change the blocking files config back down? (YOu don't mention it).

By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush.

Is this true? By default we will flush direct to stripes? If so, that is great.

For contrast, make mention of what M. C. Srivas noted a while back that you do NOT want to enable this feature if you are filling tables evenly because it will make for more i/o.

Nice.

sizeToSplit = splitPartsCount

Typo above - should be splitPartCount

This looks excellent. +1.

stack what do you think? above comment

Here's a preview... wdyt?

Introduction

Stripe compactions is an experimental feature added in HBase 0.98 which aims to improve compactions for large regions or non-uniformly distributed row keys. In order to achieve smaller and/or more granular compactions, the store files within a region are maintained separately for several row-key sub-ranges, or "stripes", of the region. The division is not visible to the higher levels of the system, so externally each region functions as before.
This feature is fully compatible with default compactions - it can be enabled for existing tables, and the table will continue to operate normally if it's disabled later.

When to use

You might want to consider using this feature if you have:

• large regions (in that case, you can get the positive effect of much smaller regions without additional memstore and region management overhead); or
• non-uniform row keys, e.g. time dimension in a key (in that case, only the stripes receiving the new keys will keep compacting - old data will not compact as much, or at all).

According to perf testing performed, in these case the read performance can improve somewhat, and the read and write performance variability due to compactions is greatly reduced. There's overall perf improvement on large, non-uniform row key regions (hash-prefixed timestamp key) over long term. All of these performance gains are best realized when table is already large. In future, the perf improvement might also extend to region splits.

How to enable

To use stripe compactions for a table or a column family, you should set its hbase.hstore.engine.class to org.apache.hadoop.hbase.regionserver.StripeStoreEngine. Due to the nature of compactions, you also need to set the blocking file count to a high number (100 is a good default, which is 10 times the normal default of 10). If changing the existing table, you should do it when it is disabled. Examples:

alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}

alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}}

create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}

Then, you can configure the other options if needed (see below) and enable the table.
To switch back to default compactions, set hbase.hstore.engine.class to nil to unset it; or set it explicitly to "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine" (this also needs to be done on a disabled table).
When you enable a large table after changing the store engine either way, a major compaction will likely be performed on most regions. This is not a problem with new tables.

How to configure

All of the settings described below are best set on table/cf level (with the table disabled first, for the settings to apply), similar to the above, e.g.

alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}}

Region and stripe sizing

Based on your region sizing, you might want to also change your stripe sizing. By default, your new regions will start with one stripe. When the stripe is too big (16 memstore flushes size), on next compaction it will be split into two stripes. Stripe splitting will continue in a similar manner as the region grows, until the region itself is big enough to split (region split will work the same as with default compactions).
You can improve this pattern for your data. You should generally aim at stripe size of at least 1Gb, and about 8-12 stripes for uniform row keys - so, for example if your regions are 30 Gb, 12x2.5Gb stripes might be a good idea.
The settings are as follows:

Memstore sizing

By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush. This approach minimizes write amplification, but can be undesirable if memstore is small and there are many stripes (the files will be too small).
In such cases, you can set hbase.store.stripe.compaction.flushToL0 to true. This will cause flush to create a single file instead; when at least hbase.store.stripe.compaction.minFilesL0 such files (by default, 4) accumulate, they will be compacted into striped files.

Normal compaction configuration

All the settings that apply to normal compactions (file size limits, etc.) apply to stripe compactions. The exception are min and max number of files, which are set to higher values by default because the files in stripes are smaller. To control these for stripe compactions, use hbase.store.stripe.compaction.minFiles and .maxFiles.