[HBASE-9854] initial documentation for stripe compactions - ASF JIRA

Hide

Permalink

Sergey Shelukhin added a comment - 08/Nov/13 00:59

Here's a preview... wdyt?

Introduction

Stripe compactions is an experimental feature added in HBase 0.98 which aims to improve compactions for large regions or non-uniformly distributed row keys. In order to achieve smaller and/or more granular compactions, the store files within a region are maintained separately for several row-key sub-ranges, or "stripes", of the region. The division is not visible to the higher levels of the system, so externally each region functions as before.
This feature is fully compatible with default compactions - it can be enabled for existing tables, and the table will continue to operate normally if it's disabled later.

When to use

You might want to consider using this feature if you have:

large regions (in that case, you can get the positive effect of much smaller regions without additional memstore and region management overhead); or
non-uniform row keys, e.g. time dimension in a key (in that case, only the stripes receiving the new keys will keep compacting - old data will not compact as much, or at all).

According to perf testing performed, in these case the read performance can improve somewhat, and the read and write performance variability due to compactions is greatly reduced. There's overall perf improvement on large, non-uniform row key regions (hash-prefixed timestamp key) over long term. All of these performance gains are best realized when table is already large. In future, the perf improvement might also extend to region splits.

How to enable

To use stripe compactions for a table or a column family, you should set its hbase.hstore.engine.class to org.apache.hadoop.hbase.regionserver.StripeStoreEngine. Due to the nature of compactions, you also need to set the blocking file count to a high number (100 is a good default, which is 10 times the normal default of 10). If changing the existing table, you should do it when it is disabled. Examples:

alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}

alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}}

create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}

Then, you can configure the other options if needed (see below) and enable the table.
To switch back to default compactions, set hbase.hstore.engine.class to nil to unset it; or set it explicitly to "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine" (this also needs to be done on a disabled table).
When you enable a large table after changing the store engine either way, a major compaction will likely be performed on most regions. This is not a problem with new tables.

How to configure

All of the settings described below are best set on table/cf level (with the table disabled first, for the settings to apply), similar to the above, e.g.

alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}}

Region and stripe sizing

Based on your region sizing, you might want to also change your stripe sizing. By default, your new regions will start with one stripe. When the stripe is too big (16 memstore flushes size), on next compaction it will be split into two stripes. Stripe splitting will continue in a similar manner as the region grows, until the region itself is big enough to split (region split will work the same as with default compactions).
You can improve this pattern for your data. You should generally aim at stripe size of at least 1Gb, and about 8-12 stripes for uniform row keys - so, for example if your regions are 30 Gb, 12x2.5Gb stripes might be a good idea.
The settings are as follows:

Setting	Notes
`hbase.store.stripe.` `initialStripeCount`	Initial stripe count to create. You can use it as follows: for relatively uniform row keys, if you know the approximate target number of stripes from the above, you can avoid some splitting overhead by starting w/several stripes (2, 5, 10...). Note that if the early data is not representative of overall row key distribution, this will not be as efficient. for existing tables with lots of data, you can use this to pre-split stripes. for e.g. hash-prefixed sequential keys, with more than one hash prefix per region, you know that some pre-splitting makes sense.
`hbase.store.stripe.` `sizeToSplit`	Maximum stripe size before it's split. You can use this in conjunction with the next setting to control target stripe size (sizeToSplit = splitPartsCount * target stripe size), according to the above sizing considerations.
`hbase.store.stripe.` `splitPartCount`	The number of new stripes to create when splitting one. The default is 2, and is good for most cases. For non-uniform row keys, you might experiment with increasing the number somewhat (3-4), to isolate the arriving updates into narrower slice of the region with just one split instead of several.

Memstore sizing

By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush. This approach minimizes write amplification, but can be undesirable if memstore is small and there are many stripes (the files will be too small).
In such cases, you can set hbase.store.stripe.compaction.flushToL0 to true. This will cause flush to create a single file instead; when at least hbase.store.stripe.compaction.minFilesL0 such files (by default, 4) accumulate, they will be compacted into striped files.

Normal compaction configuration

All the settings that apply to normal compactions (file size limits, etc.) apply to stripe compactions. The exception are min and max number of files, which are set to higher values by default because the files in stripes are smaller. To control these for stripe compactions, use hbase.store.stripe.compaction.minFiles and .maxFiles.

Show

Sergey Shelukhin added a comment - 08/Nov/13 00:59 Here's a preview... wdyt? Introduction Stripe compactions is an experimental feature added in HBase 0.98 which aims to improve compactions for large regions or non-uniformly distributed row keys. In order to achieve smaller and/or more granular compactions, the store files within a region are maintained separately for several row-key sub-ranges, or "stripes", of the region. The division is not visible to the higher levels of the system, so externally each region functions as before. This feature is fully compatible with default compactions - it can be enabled for existing tables, and the table will continue to operate normally if it's disabled later. When to use You might want to consider using this feature if you have: large regions (in that case, you can get the positive effect of much smaller regions without additional memstore and region management overhead); or non-uniform row keys, e.g. time dimension in a key (in that case, only the stripes receiving the new keys will keep compacting - old data will not compact as much, or at all). According to perf testing performed, in these case the read performance can improve somewhat, and the read and write performance variability due to compactions is greatly reduced. There's overall perf improvement on large, non-uniform row key regions (hash-prefixed timestamp key) over long term. All of these performance gains are best realized when table is already large. In future, the perf improvement might also extend to region splits. How to enable To use stripe compactions for a table or a column family, you should set its hbase.hstore.engine.class to org.apache.hadoop.hbase.regionserver.StripeStoreEngine . Due to the nature of compactions, you also need to set the blocking file count to a high number (100 is a good default, which is 10 times the normal default of 10). If changing the existing table, you should do it when it is disabled. Examples: alter 'orders_table', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} alter 'orders_table', {NAME => 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'}} create 'orders_table', 'blobs_cf', CONFIGURATION => {'hbase.hstore.engine.class' => 'org.apache.hadoop.hbase.regionserver.StripeStoreEngine', 'hbase.hstore.blockingStoreFiles' => '100'} Then, you can configure the other options if needed (see below) and enable the table. To switch back to default compactions, set hbase.hstore.engine.class to nil to unset it; or set it explicitly to " org.apache.hadoop.hbase.regionserver.DefaultStoreEngine " (this also needs to be done on a disabled table). When you enable a large table after changing the store engine either way, a major compaction will likely be performed on most regions. This is not a problem with new tables. How to configure All of the settings described below are best set on table/cf level (with the table disabled first, for the settings to apply), similar to the above, e.g. alter 'orders_table', CONFIGURATION => {'key' => 'value', ..., 'key' => 'value'}} Region and stripe sizing Based on your region sizing, you might want to also change your stripe sizing. By default, your new regions will start with one stripe. When the stripe is too big (16 memstore flushes size), on next compaction it will be split into two stripes. Stripe splitting will continue in a similar manner as the region grows, until the region itself is big enough to split (region split will work the same as with default compactions). You can improve this pattern for your data. You should generally aim at stripe size of at least 1Gb, and about 8-12 stripes for uniform row keys - so, for example if your regions are 30 Gb, 12x2.5Gb stripes might be a good idea. The settings are as follows: Setting Notes hbase.store.stripe. initialStripeCount Initial stripe count to create. You can use it as follows: for relatively uniform row keys, if you know the approximate target number of stripes from the above, you can avoid some splitting overhead by starting w/several stripes (2, 5, 10...). Note that if the early data is not representative of overall row key distribution, this will not be as efficient. for existing tables with lots of data, you can use this to pre-split stripes. for e.g. hash-prefixed sequential keys, with more than one hash prefix per region, you know that some pre-splitting makes sense. hbase.store.stripe. sizeToSplit Maximum stripe size before it's split. You can use this in conjunction with the next setting to control target stripe size (sizeToSplit = splitPartsCount * target stripe size), according to the above sizing considerations. hbase.store.stripe. splitPartCount The number of new stripes to create when splitting one. The default is 2, and is good for most cases. For non-uniform row keys, you might experiment with increasing the number somewhat (3-4), to isolate the arriving updates into narrower slice of the region with just one split instead of several. Memstore sizing By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush. This approach minimizes write amplification, but can be undesirable if memstore is small and there are many stripes (the files will be too small). In such cases, you can set hbase.store.stripe.compaction.flushToL0 to true. This will cause flush to create a single file instead; when at least hbase.store.stripe.compaction.minFilesL0 such files (by default, 4) accumulate, they will be compacted into striped files. Normal compaction configuration All the settings that apply to normal compactions (file size limits, etc.) apply to stripe compactions. The exception are min and max number of files, which are set to higher values by default because the files in stripes are smaller. To control these for stripe compactions, use hbase.store.stripe.compaction.minFiles and .maxFiles .

Hide

Permalink

Sergey Shelukhin added a comment - 08/Nov/13 01:01

stack what do you think? above comment

Show

Sergey Shelukhin added a comment - 08/Nov/13 01:01 stack what do you think? above comment

Hide

Permalink

Enis Soztutar added a comment - 08/Nov/13 02:38

This looks excellent. +1.

Show

Enis Soztutar added a comment - 08/Nov/13 02:38 This looks excellent. +1.

Hide

Permalink

Ted Yu added a comment - 08/Nov/13 23:41

Nice.

sizeToSplit = splitPartsCount

Typo above - should be splitPartCount

Show

Ted Yu added a comment - 08/Nov/13 23:41 Nice. sizeToSplit = splitPartsCount Typo above - should be splitPartCount

Hide

Permalink

stack added a comment - 11/Nov/13 22:38

Sergey Shelukhin Very nice doc especially the way you introduce the feature. Want me to add it into the doc for you?

To set it back to default, should you change the blocking files config back down? (YOu don't mention it).

By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush.

Is this true? By default we will flush direct to stripes? If so, that is great.

For contrast, make mention of what M. C. Srivas noted a while back that you do NOT want to enable this feature if you are filling tables evenly because it will make for more i/o.

Show

stack added a comment - 11/Nov/13 22:38 Sergey Shelukhin Very nice doc especially the way you introduce the feature. Want me to add it into the doc for you? To set it back to default, should you change the blocking files config back down? (YOu don't mention it). By default, the flush creates several files from one memstore, according to existing stripe boundaries and row keys to flush. Is this true? By default we will flush direct to stripes? If so, that is great. For contrast, make mention of what M. C. Srivas noted a while back that you do NOT want to enable this feature if you are filling tables evenly because it will make for more i/o.

Hide

Permalink

Sergey Shelukhin added a comment - 11/Nov/13 22:43

I will make a doc patch, hopefully tomorrow or day after.
You do want to use it for uniform tables if your regions are large, at the cost of slightly more I/O (w/flush-into-stripes) you can get more stable behavior. In my tests on multi-Gb regions reads were also slightly faster with 10-12 stripes, although I didn't investigate why then

Show

Sergey Shelukhin added a comment - 11/Nov/13 22:43 I will make a doc patch, hopefully tomorrow or day after. You do want to use it for uniform tables if your regions are large, at the cost of slightly more I/O (w/flush-into-stripes) you can get more stable behavior. In my tests on multi-Gb regions reads were also slightly faster with 10-12 stripes, although I didn't investigate why then

Hide

Permalink

Sergey Shelukhin added a comment - 14/Nov/13 21:15

Hmm... which part of doc should this go to?

Show

Sergey Shelukhin added a comment - 14/Nov/13 21:15 Hmm... which part of doc should this go to?

Hide

Permalink

Nick Dimiduk added a comment - 14/Nov/13 21:24

How about http://hbase.apache.org/book.html#compaction ?

Show

Nick Dimiduk added a comment - 14/Nov/13 21:24 How about http://hbase.apache.org/book.html#compaction ?

Hide

Permalink

Sergey Shelukhin added a comment - 14/Nov/13 22:41

Patch. I am not able to build mvn site, fails with OOM seemingly no matter what I do. But, it did catch syntax problems when there were some before going to OOM, so I assume it's mostly validated.

Show

Sergey Shelukhin added a comment - 14/Nov/13 22:41 Patch. I am not able to build mvn site, fails with OOM seemingly no matter what I do. But, it did catch syntax problems when there were some before going to OOM, so I assume it's mostly validated.

Hide

Permalink

Hadoop QA added a comment - 15/Nov/13 01:05

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12613960/HBASE-9854.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 lineLengths. The patch does not introduce lines longer than 100

-1 site. The patch appears to cause mvn site goal to fail.

+1 core tests. The patch passed unit tests in .

This message is automatically generated.

Show

Hadoop QA added a comment - 15/Nov/13 01:05 -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12613960/HBASE-9854.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 -1 site . The patch appears to cause mvn site goal to fail. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7868//console This message is automatically generated.

Hide

Permalink

stack added a comment - 25/Nov/13 19:57

Committed for Sergey Shelukhin

Show

stack added a comment - 25/Nov/13 19:57 Committed for Sergey Shelukhin

Hide

Permalink

Sergey Shelukhin added a comment - 25/Nov/13 21:58

Thanks!

Show

Sergey Shelukhin added a comment - 25/Nov/13 21:58 Thanks!

Hide

Permalink

Hudson added a comment - 26/Nov/13 03:16

FAILURE: Integrated in HBase-TRUNK #4698 (See https://builds.apache.org/job/HBase-TRUNK/4698/)
~~HBASE-9854~~ initial documentation for stripe compactions (stack: rev 1545382)

/hbase/trunk/src/main/docbkx/book.xml

Show

Hudson added a comment - 26/Nov/13 03:16 FAILURE: Integrated in HBase-TRUNK #4698 (See https://builds.apache.org/job/HBase-TRUNK/4698/ ) HBASE-9854 initial documentation for stripe compactions (stack: rev 1545382) /hbase/trunk/src/main/docbkx/book.xml

Hide

Permalink

Hudson added a comment - 26/Nov/13 11:29

SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #852 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/852/)
~~HBASE-9854~~ initial documentation for stripe compactions (stack: rev 1545382)

/hbase/trunk/src/main/docbkx/book.xml

Show

Hudson added a comment - 26/Nov/13 11:29 SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #852 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/852/ ) HBASE-9854 initial documentation for stripe compactions (stack: rev 1545382) /hbase/trunk/src/main/docbkx/book.xml

Hide

Permalink

Lars Francke added a comment - 20/Nov/15 11:54

This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

Show

Lars Francke added a comment - 20/Nov/15 11:54 This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

initial documentation for stripe compactions

Details

Description

Attachments

Activity

Introduction

When to use

How to enable

How to configure

Region and stripe sizing

Memstore sizing

Normal compaction configuration

People

Dates

Development

Agile