[HBASE-8566] Master-Slave replication: truncate action isn't sent over to slave cluster and cause data inconsistency

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 0.94.3
• Component/s: Replication

Environment

two 2-nodes clusters setup as Master and Slave for replication of table 'usertable'

• Assignee: Unassigned
• Reporter: Demai Ni
• Created: 16/May/13 22:37
• Updated: 16/Jun/22 16:54
• Resolved: 16/Jun/22 16:54

Description

after successfully setup the replcation. put some rows into 'usertable' ,

At Master cluster
$truncate 'usertable'

The truncate(or mass delete from user perspective) request isn't sent over to slave cluster.

From internal, the truncate is 'disable', 'drop' and 'create'. Such operations are not designed for replication. However, from external/user perspective, this is a 'delete everything' operation, which should be part of the replication.

This JIRA is to add this support
---------------------------
additional information. I did a few loads using YCSB into 'usertable', with different # of rows(from 1000 to 100000). And did truncate a couple times in between. Then the slave cluster began to throw errors:

count failed on slave cluster

ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions:
Thu May 16 15:00:13 PDT 2013, org.apache.hadoop.hbase.client.ScannerCallable@4c474c47, java.net.ConnectException: Connection refused
Thu May 16 15:00:32 PDT 2013, org.apache.hadoop.hbase.client.ScannerCallable@4c474c47, org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for usertable,,99999999999999 after 7 tries.
Thu May 16 15:00:51 PDT 2013, org.apache.hadoop.hbase.client.ScannerCallable@4c474c47, org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for usertable,,99999999999999 after 7 tries.
Thu May 16 15:01:11 PDT 2013, org.apache.hadoop.hbase.client.ScannerCallable@4c474c47, org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find region for usertable,,99999999999999 after 7 tries.

The regionserver log of slave cluster throws :

regionserver log of slave cluster

2013-05-16 14:59:59,655 ERROR org.apache.hadoop.hbase.replication.regionserver.ReplicationSink: Unable to accept edit because:
java.io.IOException: java.lang.InterruptedException
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:220)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.replicateEntries(ReplicationSink.java:154)
        at org.apache.hadoop.hbase.replication.regionserver.Replication.replicateLogEntries(Replication.java:140)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.replicateLogEntries(HRegionServer.java:3797)
        at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Caused by: java.lang.InterruptedException
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:853)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1507)
        at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1400)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:699)
        at org.apache.hadoop.hbase.replication.regionserver.ReplicationSink.batch(ReplicationSink.java:217)
        ... 8 more

Issue Links

• is related to: HBASE-8332 Add truncate as HMaster method