Details
New Feature
Description
DelimitedKeyPrefixRegionSplitPolicy similar to KeyPrefixRegionSplitPolicy, but with a delimiter for the key, instead of a fixed prefix.
Can be used for META regions, since we are doing table_name,start_key,region_id.encoded_region_name.
Issue Links
- is related to
-
HBASE-8856
Crash when attempting split with DelimitedKeyPrefixRegionSplitPolicy
-
- Closed
-
-
HBASE-7877
KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy splits are not always optimal
-
- Open
-
-
HBASE-7721
Atomic multi-row mutations in META
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Patch v2. Change the configuration key for KeyPrefixRegionSplitPolicy
+1 on patch
+1 on idea
+1 on using for atomic META transactions
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12567704/hbase-7748_v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 lineLengths. The patch introduces lines longer than 100
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitLogWorker
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/4301//console
This message is automatically generated.
+ * I.e. rows can be co-located in a regionb by their prefix.
Typo: regionb
+ * As and example, if you have row keys delimited with <code>_</code>, like
Typo: and
+ * ensures that all rows staring with the same userid, belongs to the same region.
Typo: staring, belongs
Please add stability annotation.
+ //find the first occurrence of delimiter in split point
The above comment should be explicitly mentioned in class javadoc where delimiter appears twice in rowkey:
+ * <code>userid_eventtype_eventid</code>, and use prefix delimiter _, this split policy
+1 on patch (edit javadoc on commit)
Committed this with Ted's suggestions. Will also commit this in 0.94.6 unless objection.
Attaching committed version, v3.
Integrated in HBase-TRUNK #3849 (See https://builds.apache.org/job/HBase-TRUNK/3849/)
HBASE-7748. Add DelimitedKeyPrefixRegionSplitPolicy (Revision 1442408)
Result = FAILURE
enis :
Files :
- /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
- /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java
- /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #391 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/391/)
HBASE-7748. Add DelimitedKeyPrefixRegionSplitPolicy (Revision 1442408)
Result = FAILURE
enis :
Files :
- /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
- /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java
- /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
Committed this to 0.94 branch. Attaching straightforward port.
Integrated in HBase-0.94 #824 (See https://builds.apache.org/job/HBase-0.94/824/)
HBASE-7748. Add DelimitedKeyPrefixRegionSplitPolicy (Revision 1442803)
Result = SUCCESS
enis :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
Integrated in HBase-0.94-security #107 (See https://builds.apache.org/job/HBase-0.94-security/107/)
HBASE-7748. Add DelimitedKeyPrefixRegionSplitPolicy (Revision 1442803)
Result = FAILURE
enis :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
Am I correct in assuming that if there is a very uneven distribution of user's, and the region has say 5 users, but the first user in the region takes up over half the space (and thus the split point picks that user) that this implementation will wind up not splitting?
Wouldn't it make sense to find the row group's first row (what the current patch does) and find the next group's first row and choose based on which one gives a better split?
@Robert:
Good observation.
@Robert, that looks like a corner case, which should be handled by changing the data model, rather than split policy, no?
@Enis, perhaps a change in the data model would avoid this situation. However to me, regardless of the data model, it appears that this behaviour is non-optimal.
We select a split point (roughly the middle) and then arbitrarily move it one direction (to find a group boundary). The original split point is the most optimal, in terms of splitting. Thus, we should find the nearest usable split point to that row and maintain as optimal a split as possible.
Sure in the example I gave it is an extreme case, but even ignoring that you might end up with non-optimal splits. It may be the case that moving down 1 single row would find a group boundary, yet we move up back rows anyway.
BTW all, I filed HBASE-7877 to fix this inefficiency.
Integrated in HBase-0.94-security-on-Hadoop-23 #12 (See https://builds.apache.org/job/HBase-0.94-security-on-Hadoop-23/12/)
HBASE-7748. Add DelimitedKeyPrefixRegionSplitPolicy (Revision 1442803)
Result = FAILURE
enis :
Files :
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java
- /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionSplitPolicy.java
Attaching patch.