This issue was closed as part of a bulk closing operation on 2015-11-20. All issues that have been resolved and where all fixVersions have been released have been closed (following discussions on the mailing list).

SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #814 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/814/)
HBASE-8000 create integration/perf tests for stripe compactions (sershe: rev 1536571)

• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngest.java
• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestStripeCompactions.java
• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/StripeCompactionsPerformanceEvaluation.java
• /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java

SUCCESS: Integrated in HBase-TRUNK #4651 (See https://builds.apache.org/job/HBase-TRUNK/4651/)
HBASE-8000 create integration/perf tests for stripe compactions (sershe: rev 1536571)

• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngest.java
• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestStripeCompactions.java
• /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/StripeCompactionsPerformanceEvaluation.java
• /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java

Committed to trunk.

v3 looks good.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12604908/HBASE-8000-latest-with-dependencies.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 25 new or modified tests.

+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

-1 javadoc. The javadoc tool appears to have generated 7 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

-1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 lineLengths. The patch does not introduce lines longer than 100

+1 site. The mvn site goal succeeds with this patch.

+1 core tests. The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/7365//console

This message is automatically generated.

rebase for new integration test approach; move non-CM tests into perf tool rather than integration tests

rewrote the tests based in HBASE-8598 to enable more realistic perf testing with overwrites, deletes and measuring the perf of random reads, not latest reads as is the normal LoadTestTool behavior

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12580855/HBASE-8000-v1-with-7679-7680-7967.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 39 new or modified tests.

+1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

-1 javadoc. The javadoc tool appears to have generated 7 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

-1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

-1 lineLengths. The patch introduces lines longer than 100

+1 site. The mvn site goal succeeds with this patch.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.TestHFileArchiving

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//testReport/
Release audit warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5479//console

This message is automatically generated.

rebase patch

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12576455/HBASE-8000-v0-with-7679-7680-7967.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 39 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

-1 javadoc. The javadoc tool appears to have generated 7 warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 lineLengths. The patch introduces lines longer than 100

+1 site. The mvn site goal succeeds with this patch.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog

-1 core zombie tests. There are 1 zombie test(s):

Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5086//console

This message is automatically generated.

https://reviews.apache.org/r/10230/

Proper patch

Temporary patch for the document context... it is close to complete though.

count-based

Thanks for the update, Sergey

Were you exercising count-based stripes, size-based stripes or both ?

Preliminary results:
1) I found some efficiency of selection issues in stripe policy, updated the respective patch... a few tiny bugs too, mostly in HStore using the thing.
2) Haven't tested after fixing (1), but even before I can see the observed pattern that stripe compaction is slightly slower on average with large data due to constant small compactions, but doesn't have huge latency spikes that default policy creates. I'll see how it can be tweaked, and how much of the overall slowdown will be fixed by fixing (1).
3) Now the sad part. This can only be evaluated on actual large data (unless somebody knows some tricks to make Hadoop really slow or otherwise test in such cases?). I have to run it for a few hours on EC2.

For 94 backport I was talking only about the performance evaluation, not the stripe compactions itself. To compare major compactions or similar processes between 0.94 where we don't have the stripe compaction, and 0.96 where we have it. This might allow us to figure how stripe compactions is impacting the performances.

Adding to performance evaluation is possible; integration test would be nice first to see that it works beyond unit tests.
As for 94, it depends on many previous refactoring (intrusive) JIRAs so that's very unlikely.

Sergey Shelukhin will it be possible to add the perf test in the existing class? Like adding it to PerformanceEvaluation or LoadTestTool? Also, will be good if this can be backport to a 0.94 release where the stripe compaction is not there, to compare the performances with and without the strip compaction activated.