Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.3.0
-
None
-
Reviewed
Description
Currently the support for replication of bulk loaded hfiles relies on bulk load marker written in the WAL.
The move of bulk loaded hfile(s) (into region directory) may succeed but the write of bulk load marker may fail.
This means that although bulk loaded hfile is being served in source cluster, the replication wouldn't happen.
Normally operator is supposed to retry the bulk load. But relying on human retry is not robust solution.
Attachments
Attachments
- HBASE-17290.patch
- 24 kB
- Ashish Singhi
- HBASE-17290.v1.patch
- 27 kB
- Ashish Singhi
- HBASE-17290.branch-1.patch
- 25 kB
- Ashish Singhi
Issue Links
- is related to
-
HBASE-13153 Bulk Loaded HFile Replication
- Closed
Activity
HBASE-15446 is still open.
As the description of HBASE-15446 says, the solution depends on client to retry. What if client doesn't retry ?
See the latest patch on HBASE-14417 for solution - we record the hfile before the movement into region directory using the hook introduced by HBASE-17292.
Sorry for the delay, got stuck with company work.
I have attached the patch.
Added a new RS observer, ReplicationObserver to solve this bug.
Please review.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 13s | Docker mode activated. |
+1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
0 | mvndep | 0m 23s | Maven dependency ordering for branch |
+1 | mvninstall | 3m 6s | master passed |
+1 | compile | 0m 49s | master passed |
+1 | checkstyle | 0m 42s | master passed |
+1 | mvneclipse | 0m 23s | master passed |
+1 | findbugs | 2m 29s | master passed |
+1 | javadoc | 0m 43s | master passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 0m 56s | the patch passed |
+1 | compile | 0m 50s | the patch passed |
+1 | javac | 0m 50s | the patch passed |
+1 | checkstyle | 0m 42s | the patch passed |
+1 | mvneclipse | 0m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | hadoopcheck | 26m 35s | Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. |
+1 | findbugs | 2m 46s | the patch passed |
+1 | javadoc | 0m 43s | the patch passed |
+1 | unit | 1m 5s | hbase-client in the patch passed. |
-1 | unit | 81m 7s | hbase-server in the patch failed. |
+1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
125m 5s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hbase.replication.TestMasterReplication |
hadoop.hbase.regionserver.TestHRegion |
Subsystem | Report/Notes |
---|---|
Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12845785/HBASE-17290.patch |
JIRA Issue | |
Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
uname | Linux ef0fbfb7688c 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
git revision | master / dba103e |
Default Java | 1.8.0_111 |
findbugs | v3.0.0 |
unit | https://builds.apache.org/job/PreCommit-HBASE-Build/5145/artifact/patchprocess/patch-unit-hbase-server.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/5145/artifact/patchprocess/patch-unit-hbase-server.txt |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5145/testReport/ |
modules | C: hbase-client hbase-server U: . |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5145/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Test failure was caused by:
Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.replication.TestMasterReplication.startMiniClusters(TestMasterReplication.java:492) at org.apache.hadoop.hbase.replication.TestMasterReplication.testHFileMultiSlaveReplication(TestMasterReplication.java:331)
Please fix in the next patch.
Please enrich the javadoc with explanation for the second component of the Pair.
+ * @param pairs list of hfile references to be added
Please make the parameter name consistent - should be pairs
+ public void addHFileRefs(String peerId, List<Pair<Path, Path>> files)
For ReplicationObserver.java, please reference the format of license header of existing classes.
Consider adding @InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.CONFIG) to this class.
+ LOG.debug("Skipping recording bulk load entries in preCommitStoreFile for bulkloaded " + + "data replication.");
It would be better if the case for bulk load replication and the case where pairs is empty are logged separately.
This would facilitate troubleshooting.
Can you point me to the code which handles the case where Path for bulk loaded hfile is recorded but the commit (move of hfile) fails ?
In that scenario, the file wouldn't be found at time of replication.
Can you point me to the code which handles the case where Path for bulk loaded hfile is recorded but the commit (move of hfile) fails ?
In that scenario, the file wouldn't be found at time of replication.
In that scenario we will end up here, https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java#L380
The LOG in that catch block is at error level.
Consider changing it to DEBUG since the absence of hfile implies error in the commit step (which should be remedied by operator retry).
+1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 10s | Docker mode activated. |
+1 | hbaseanti | 0m 0s | Patch does not have any anti-patterns. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
0 | mvndep | 0m 10s | Maven dependency ordering for branch |
+1 | mvninstall | 2m 51s | master passed |
+1 | compile | 0m 50s | master passed |
+1 | checkstyle | 0m 42s | master passed |
+1 | mvneclipse | 0m 21s | master passed |
+1 | findbugs | 2m 29s | master passed |
+1 | javadoc | 0m 43s | master passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 0m 57s | the patch passed |
+1 | compile | 0m 51s | the patch passed |
+1 | javac | 0m 51s | the patch passed |
+1 | checkstyle | 0m 42s | the patch passed |
+1 | mvneclipse | 0m 22s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | hadoopcheck | 26m 4s | Patch does not cause any errors with Hadoop 2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. |
+1 | findbugs | 2m 47s | the patch passed |
+1 | javadoc | 0m 43s | the patch passed |
+1 | unit | 1m 1s | hbase-client in the patch passed. |
+1 | unit | 80m 47s | hbase-server in the patch passed. |
+1 | asflicense | 0m 26s | The patch does not generate ASF License warnings. |
123m 36s |
Subsystem | Report/Notes |
---|---|
Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12845828/HBASE-17290.v1.patch |
JIRA Issue | |
Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
uname | Linux 45b15a51594d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh |
git revision | master / df98d8d |
Default Java | 1.8.0_111 |
findbugs | v3.0.0 |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5149/testReport/ |
modules | C: hbase-client hbase-server U: . |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5149/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Planning to commit this after an hour or so, if there is no further review comments.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 10m 37s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. |
0 | mvndep | 0m 21s | Maven dependency ordering for branch |
+1 | mvninstall | 1m 42s | branch-1 passed |
+1 | compile | 0m 44s | branch-1 passed with JDK v1.8.0_111 |
+1 | compile | 0m 50s | branch-1 passed with JDK v1.7.0_80 |
+1 | checkstyle | 0m 52s | branch-1 passed |
+1 | mvneclipse | 0m 28s | branch-1 passed |
-1 | findbugs | 1m 54s | hbase-server in branch-1 has 2 extant Findbugs warnings. |
+1 | javadoc | 0m 39s | branch-1 passed with JDK v1.8.0_111 |
+1 | javadoc | 0m 52s | branch-1 passed with JDK v1.7.0_80 |
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 0m 59s | the patch passed |
+1 | compile | 0m 44s | the patch passed with JDK v1.8.0_111 |
+1 | javac | 0m 44s | the patch passed |
+1 | compile | 0m 51s | the patch passed with JDK v1.7.0_80 |
-1 | javac | 0m 34s | hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 1 new + 4 unchanged - 1 fixed = 5 total (was 5) |
+1 | checkstyle | 0m 52s | the patch passed |
+1 | mvneclipse | 0m 27s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | hadoopcheck | 14m 57s | The patch does not cause any errors with Hadoop 2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. |
+1 | hbaseprotoc | 0m 25s | the patch passed |
+1 | findbugs | 3m 20s | the patch passed |
+1 | javadoc | 0m 39s | the patch passed with JDK v1.8.0_111 |
+1 | javadoc | 0m 53s | the patch passed with JDK v1.7.0_80 |
+1 | unit | 1m 50s | hbase-client in the patch passed. |
-1 | unit | 84m 50s | hbase-server in the patch failed. |
+1 | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
132m 14s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hbase.client.TestMetaWithReplicas |
Subsystem | Report/Notes |
---|---|
Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e01ee2f |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12845959/HBASE-17290.branch-1.patch |
JIRA Issue | |
Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck hbaseanti checkstyle compile |
uname | Linux 9dd280c94211 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
git revision | branch-1 / 667c5eb |
Default Java | 1.7.0_80 |
Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-oracle:1.7.0_80 |
findbugs | v3.0.0 |
findbugs | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html |
javac | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/artifact/patchprocess/diff-compile-javac-hbase-server-jdk1.7.0_80.txt |
unit | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/artifact/patchprocess/patch-unit-hbase-server.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/artifact/patchprocess/patch-unit-hbase-server.txt |
Test Results | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/testReport/ |
modules | C: hbase-client hbase-server U: . |
Console output | https://builds.apache.org/job/PreCommit-HBASE-Build/5158/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Failed junit tests hadoop.hbase.client.TestMetaWithReplicas
Test failure was not related to patch. It is failing on other build runs also here, here...
FAILURE: Integrated in Jenkins build HBase-1.4 #585 (See https://builds.apache.org/job/HBase-1.4/585/)
HBASE-17290 Potential loss of data for replication of bulk loaded hfiles (ashishsinghi: rev e8e40d86258a2f19838f3aea55dafbbc1b860942)
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2269 (See https://builds.apache.org/job/HBase-Trunk_matrix/2269/)
HBASE-17290 Potential loss of data for replication of bulk loaded hfiles (ashishsinghi: rev 5f631b9653a4bf86a2bebed58abed747c04b704f)
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/TableBasedReplicationQueuesImpl.java
- (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #291 (See https://builds.apache.org/job/HBase-1.3-JDK7/291/)
HBASE-18900 Backport HBASE-17290 "Potential loss of data for replication (tedyu: rev 8e5f5dd655d6f16c0d9f421c308a05ce5fa61c72)
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #219 (See https://builds.apache.org/job/HBase-1.3-IT/219/)
HBASE-18900 Backport HBASE-17290 "Potential loss of data for replication (tedyu: rev 8e5f5dd655d6f16c0d9f421c308a05ce5fa61c72)
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #303 (See https://builds.apache.org/job/HBase-1.3-JDK8/303/)
HBASE-18900 Backport HBASE-17290 "Potential loss of data for replication (tedyu: rev 8e5f5dd655d6f16c0d9f421c308a05ce5fa61c72)
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (add) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
SUCCESS: Integrated in Jenkins build HBase-1.3-IT #230 (See https://builds.apache.org/job/HBase-1.3-IT/230/)
HBASE-18900 Backport HBASE-17290 - revert due to incompatibility (tedyu: rev ccf9b85d040e9028bcab0d49321be104d67fc76e)
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #302 (See https://builds.apache.org/job/HBase-1.3-JDK7/302/)
HBASE-18900 Backport HBASE-17290 - revert due to incompatibility (tedyu: rev ccf9b85d040e9028bcab0d49321be104d67fc76e)
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #316 (See https://builds.apache.org/job/HBase-1.3-JDK8/316/)
HBASE-18900 Backport HBASE-17290 - revert due to incompatibility (tedyu: rev ccf9b85d040e9028bcab0d49321be104d67fc76e)
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
- (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
- (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java
- (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java
- (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationObserver.java
Sorry for asking it too late.
When write marker cell in WAL is failed, do we succed the bulk load op still?
On the new CP, we use post commit hook of store file. So this will be called for all files including the flushed ones right. So when this CP is in place, how will we treat diff btw bulk loaded files and normal flushed/compaction result files?
The CP seems to listen for the post commit hook and then add an entry into the hfiles for replication Q. I wonder why we cannot do this in core code flow itself. WHy we need an extra CP? This is not a sync HFile replication op. Just adding an entry. Said that even the WAL write also can be avoided?
Really sorry for asking this late after its commit. Got to see this code while doing the recent CP cleanups only.
I believe this point was discussed in the original feature issue.. Sorry I totally forgot abt it now.. ashish singhi on vacation right now and once he is back, he can comment abt the discussion points happened at that time.