Details
Description
On HBASE-12254 and also at https://github.com/risdenk/hbase-custom-replication-endpoint-example risdenk made the following observations and suggestions regarding custom replication endpoints that I think are a reasonable blueprint for improvement:
I was trying out the pluggable replication endpoint feature and found the following:
- you must use the ReplicationAdmin to add the new ReplicationEndpoint
- hbase shell add_peer command doesn't support specifying a custom class
- hbase shell add_peer relies on the newly deprecated ReplicationAdmin addPeer methods
- ReplicationAdmin addPeer tableCfs is now a Map<TableName, ? extends Collection<String>> instead of a string
We should fix the add_peer command in the shell at least.
Attachments
Attachments
- HBASE-12867.patch
- 6 kB
- Kevin Risden
- HBASE-12867-v1.patch
- 8 kB
- Kevin Risden
- HBASE-12867-v2.patch
- 15 kB
- Kevin Risden
- HBASE-12867-v3.patch
- 16 kB
- Enis Soztutar
- HBASE-12867-v3.patch
- 16 kB
- Enis Soztutar
- HBASE-12867-v3.patch
- 16 kB
- Kevin Risden
Issue Links
- breaks
-
HBASE-13746 list_replicated_tables command is not listing table in hbase shell.
- Closed
- is related to
-
HBASE-12940 Expose listPeerConfigs and getPeerConfig to the HBase shell
- Closed
Activity
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12693208/HBASE-12867.patch
against master branch at commit f71b980d3715c94478930b614e63250ebb678899.
ATTACHMENT ID: 12693208
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
+1 core tests. The patch passed unit tests in .
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12514//console
This message is automatically generated.
Thanks Kevin for working on this issue.
The patch adds a new command called "add_custom_peer" which takes an id, replicationEndpointImpl, and tableCFs.
I think we don't need to add a new command. Current add_peer command should be overwritable.
My understanding is that the clusterKey is only for remote clusters for HBase replication so for adding a custom replication endpoint the clusterKey can be pulled from the HBase configuration.
Yes, clusterKey is a custom form of zk quorum address + parent znode that is used for intercluster replication. Similar settings should be settable for a custom replication endpoint implementation. For example, a replication endpoint for SOLR might get the solr cloud URL from it's own configuration. That is why the ReplicationPeerConfig class has a custom configuration field.
I wasn't sure why those addPeer methods were deprecated so didn't change them.
They are mostly deprecated because string parsing of custom table and CFs to replicate is very fragile and should not be exposed to users. Rather the API should be a set of table and column families. If we can somehow extend that to the shell (using native jruby concepts) that would be awesome.
If we can get the shell to support a hypothetical endpoint like this, it would be great:
shell > add_peer 'my-peer-id', 'org.apache.hadoop.hbase.MyReplicationEndpoint', DATA => { "key1" => 1 }, CONFIG => { "config1" => "value1, "config2" => "value2" }, TABLE_CFS => { "table1', "table2", "table3 => {"cf1", "cf2"} }
Where DATA, CONFIG and TABLE_CFS are optional.
Also, the help for add_peer mentions that the id must be a short which is not the case anymore.
I think we don't need to add a new command. Current add_peer command should be overwritable.
I originally thought that too but if we want to ensure that the add_peer command is backwards compatible for HBase 0.98 then there would need to be a new command or potentially used a named argument for the classname. Here is a simple example:
shell > add_peer 'my-peer-id', 'org.apache.hadoop.hbase.MyReplicationEndpoint'
Since DATA, CONFIG, and TABLE_CFS are optional that would just become an id and a classname. The current add_peer implementation assumes that the second argument string is the clusterKey. Without doing string parsing, it would be impossible to tell if the second argument is either a classname or a clusterKey. So I think that would make overwriting add_peer difficult if we need to support backwards compatibility.
A related thought that I can look into is if there is support for a named parameter in jruby. This would allow the appearance of add_peer being overwritten and still maintain the backwards compatibility of add_peer without ending up with the example shown above. This would require either the clusterKey to be specified or the CLASSNAME. Here are two examples:
shell > add_peer 'my-peer-id', CLASSNAME = 'org.apache.hadoop.hbase.MyReplicationEndpoint'
shell > add_peer 'my-peer-id', CLASSNAME = 'org.apache.hadoop.hbase.MyReplicationEndpoint', DATA => { "key1" => 1 }, CONFIG => { "config1" => "value1", "config2" => "value2" }, TABLE_CFS => { "table1", "table2", "table3" => {"cf1", "cf2"} }
Maybe backwards compatibility isn't required for HBase 2.x, but for 1.x and 0.98.x there should be a way to add a custom replication endpoint peer from the shell and still be able to use the add_peer command how it has been used in the past.
Also, the help for add_peer mentions that the id must be a short which is not the case anymore.
Ok the documentation was copied from add_peer so that can be changed.
but for 1.x and 0.98.x there should be a way to add a custom replication endpoint peer from the shell and still be able to use the add_peer command how it has been used in the past
Agreed. BC is needed. If adding ENDPOINT_CLASSNAME or something similar is doable in jruby, I think that is acceptable.
Added HBASE-12867-v1.patch which addresses the comments by enis.
The backwards compatibility is still there and makes use of some rubyism for it to work. The documentation for the command was updated with the only real change from the comments above being that arguments must be preceded by a colon (ie: :ENDPOINT_CLASSNAME or :TABLE_CFS).
There probably should be tests for this but didn't see tests for the other shell commands. I can look into that later this week.
Cool. I'll do a review.
There probably should be tests for this but didn't see tests for the other shell commands. I can look into that later this week.
Tests for jruby code is under hbase-shell/src/test/ruby/. There should be similar tests for shell commands to check syntax, etc. Java TestShell code drives the jruby tests. https://hbase.apache.org/book.html#hbase.tests contains some details.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12694808/HBASE-12867-v1.patch
against master branch at commit f7df0990c2d321cffd7ea2e20cb7b280d8cc9db6.
ATTACHMENT ID: 12694808
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
-1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//artifact/patchprocess/patchJavadocWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12602//console
This message is automatically generated.
Added tests for the ReplicationAdmin commands. There were no existing tests for replication_admin.rb so added a new file for them.
Through this process found that the ReplicationPeerConfig methods in ReplicationAdmin are not all exposed through the hbase shell or through replication_admin.rb. This makes testing for add_peer hard because its impossible to get back details about the peer without access to listPeerConfigs and getPeerConfig in replication_admin.rb. Should there be a new JIRA opened to address this?
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12694897/HBASE-12867-v2.patch
against master branch at commit 6b20a0f831b195c6f136a3decee61f1d80ebde29.
ATTACHMENT ID: 12694897
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12612//console
This message is automatically generated.
Kevin the patch is great.
Through this process found that the ReplicationPeerConfig methods in ReplicationAdmin are not all exposed through the hbase shell or through replication_admin.rb. This makes testing for add_peer hard because its impossible to get back details about the peer without access to listPeerConfigs and getPeerConfig in replication_admin.rb. Should there be a new JIRA opened to address this?
We can do it here or in a follow up issue. Both are ok.
In the patch why did you have a test that is commented out?
+ # assert_raise fails on native exceptions - https://jira.codehaus.org/browse/JRUBY-5279
One last thing is that, regular commands like create table, etc define constants used in hbase-shell/src/main/ruby/hbase.rb. so that the user does not have to do :CONSTANT, and instead can do CONSTANT. We can do the same thing for consistency with the other commands.
I'll create a follow up JIRA for adding listPeerConfigs and getPeerConfig to the HBase shell.
In the patch why did you have a test that is commented out?
+ # assert_raise fails on native exceptions - https://jira.codehaus.org/browse/JRUBY-5279
I wanted to test that when adding a peer with the same id that an exception is thrown up to the shell. The exception is actually a java.lang.IllegalArgumentException but I can't catch it with assert_raise. JRUBY-5279 identifies the issue and it looks like it is fixed in JRuby 1.7 but HBase is using JRuby 1.6.8. I can remove the test if that helps.
One last thing is that, regular commands like create table, etc define constants used in hbase-shell/src/main/ruby/hbase.rb. so that the user does not have to do :CONSTANT, and instead can do CONSTANT. We can do the same thing for consistency with the other commands.
Ah that makes sense. I was trying to find how the other commands didn't need to use colons. I'll make this change.
Addressed :CONSTANT -> CONSTANT by adding the following constants to hbase.rb.
ENDPOINT_CLASSNAME = 'ENDPOINT_CLASSNAME' CLUSTER_KEY = 'CLUSTER_KEY' TABLE_CFS = 'TABLE_CFS' CONFIG = 'CONFIG' DATA = 'DATA'
The outstanding item is the commented out test in replication_admin_test.rb and if that needs to be deleted or can be kept.
I wanted to test that when adding a peer with the same id that an exception is thrown up to the shell. The exception is actually a java.lang.IllegalArgumentException but I can't catch it with assert_raise. JRUBY-5279 identifies the issue and it looks like it is fixed in JRuby 1.7 but HBase is using JRuby 1.6.8. I can remove the test if that helps.
Usually, we do not commit commented out code, but in this case, we can keep the test. If we ever move to a newer version of jruby, we can enable the test later.
I'll commit if hadoopqa passes.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695140/HBASE-12867-v3.patch
against master branch at commit 6b20a0f831b195c6f136a3decee61f1d80ebde29.
ATTACHMENT ID: 12695140
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
-1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12628//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12695366/HBASE-12867-v3.patch
against master branch at commit 6bfa8ea977e991c372c0bef7faa49d6142befebc.
ATTACHMENT ID: 12695366
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
-1 core zombie tests. There are 1 zombie test(s): at org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStartedServerIsAlive(TestHttpServerLifecycle.java:71)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12635//console
This message is automatically generated.
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12696053/HBASE-12867-v3.patch
against master branch at commit fef78acf6534f6736c10d2054cf8cb479edd3291.
ATTACHMENT ID: 12696053
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 checkstyle. The applied patch does not increase the total number of checkstyle errors
+1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 lineLengths. The patch does not introduce lines longer than 100
+1 site. The mvn site goal succeeds with this patch.
-1 core tests. The patch failed these unit tests:
-1 core zombie tests. There are 2 zombie test(s): at org.apache.hadoop.test.GenericTestUtils.waitFor(GenericTestUtils.java:162)
at org.apache.hadoop.hdfs.TestDistributedFileSystem.testGetFileBlockStorageLocationsError(TestDistributedFileSystem.java:852)
at org.apache.hadoop.hbase.TestAcidGuarantees.testScanAtomicity(TestAcidGuarantees.java:354)
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//artifact/patchprocess/checkstyle-aggregate.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/12671//console
This message is automatically generated.
FAILURE: Integrated in HBase-1.1 #173 (See https://builds.apache.org/job/HBase-1.1/173/)
HBASE-12867 Add ability to specify custom replication endpoint to add_peer (enis: rev 4797b025fb73aef44f0e3d1498a669ba746af83e)
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/test/ruby/test_helper.rb
I've committed this. Thanks Kevin for the patch. I've opened up a subtask for the 0.98 backport since the unit test is failing due to lack of shutdown method (may need some more backport)
FAILURE: Integrated in HBase-1.0 #734 (See https://builds.apache.org/job/HBase-1.0/734/)
HBASE-12867 Add ability to specify custom replication endpoint to add_peer (enis: rev 6e78ab4af3924366cfc707d1642f6a200b347b18)
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
FAILURE: Integrated in HBase-TRUNK #6125 (See https://builds.apache.org/job/HBase-TRUNK/6125/)
HBASE-12867 Add ability to specify custom replication endpoint to add_peer (enis: rev 7561ae6d1257b51c0bb1ef46e52d8ede2c7c926f)
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
Reverted from 1.0.0 so can cut RC. No obvious fix after looking at it for a few minutes (nosuchmethod in generated pbs – I tried regening pbs but that was not it). TestShell is failing in branch-1.0. https://builds.apache.org/job/HBase-1.0/741/testReport/junit/org.apache.hadoop.hbase.client/TestShell/testRunShellTests/
SUCCESS: Integrated in HBase-1.0 #742 (See https://builds.apache.org/job/HBase-1.0/742/)
Revert "HBASE-12867 Add ability to specify custom replication endpoint to add_peer" (stack: rev 581770780bcbf600eaf262c0347b33b0d238ce9b)
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
Removed 1.0.1 from fixVersion due to revert. Hopefully in subtask we will re-commit to 0.98 and 1.0 after we get unit tests running.
risdenk It was only reverted from 1.0. We still have the patch in branch-1 and master. Resolving this again. We can commit it with the fixed test in https://issues.apache.org/jira/browse/HBASE-13035 to 0.98 and 1.0 branches.
FAILURE: Integrated in HBase-1.0 #916 (See https://builds.apache.org/job/HBase-1.0/916/)
HBASE-13035 Backport HBASE-12867 Shell does not support custom replication endpoint specification (apurtell: rev 31fb058394f70f1742827ce3bdc97695fb3eae81)
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
- hbase-shell/src/main/ruby/hbase.rb
FAILURE: Integrated in HBase-0.98 #990 (See https://builds.apache.org/job/HBase-0.98/990/)
HBASE-13035 Backport HBASE-12867 Shell does not support custom replication endpoint specification (apurtell: rev 4ff7797f3a754b1cbb7e6de6c78c356321ba2396)
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #941 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/941/)
HBASE-13035 Backport HBASE-12867 Shell does not support custom replication endpoint specification (apurtell: rev 4ff7797f3a754b1cbb7e6de6c78c356321ba2396)
- hbase-shell/src/main/ruby/hbase/replication_admin.rb
- hbase-shell/src/test/ruby/hbase/replication_admin_test.rb
- hbase-shell/src/test/ruby/test_helper.rb
- hbase-shell/src/main/ruby/hbase.rb
- hbase-shell/src/main/ruby/shell/commands/add_peer.rb
The patch adds a new command called "add_custom_peer" which takes an id, replicationEndpointImpl, and tableCFs.
My understanding is that the clusterKey is only for remote clusters for HBase replication so for adding a custom replication endpoint the clusterKey can be pulled from the HBase configuration.
addPeer from the HBase shell is still relying on @Deprecated methods in ReplicationAdmin. I wasn't sure why those addPeer methods were deprecated so didn't change them.