[HBASE-6533] [replication] replication will block if WAL compress set differently in master and slave configuration

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.94.0
• Component/s: Replication
• Assignee: Unassigned
• Reporter: terry zhang
• Created: 09/Aug/12 07:26
• Updated: 13/Jun/22 19:01
• Resolved: 18/Sep/12 15:25

Description

as we know in hbase 0.94.0 we have a configuration below
<property>
<name>hbase.regionserver.wal.enablecompression</name>
<value>true</value>
</property>
if we enable it in master cluster and disable it in slave cluster . Then replication will not work. It will throw unwrapRemoteException again and again in master cluster.

2012-08-09 12:49:55,892 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't replicate because of an error
on the remote cluster:
java.io.IOException: IPC server unable to read call parameters: Error in readFields
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:635)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:365)
Caused by: org.apache.hadoop.ipc.RemoteException: IPC server unable to read call parameters: Error in readFields
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:921)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:151)
at $Proxy13.replicateLogEntries(Unknown Source)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:616)
... 1 more

This is because Slave cluster can not parse the hlog entry .

2012-08-09 14:46:05,891 WARN org.apache.hadoop.ipc.HBaseServer: Unable to read call parameters for client 10.232.98.89
java.io.IOException: Error in readFields
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:685)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:635)
at org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1292)
at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1207)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:735)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:524)
at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:499)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at org.apache.hadoop.hbase.KeyValue.readFields(KeyValue.java:2254)
at org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFields(WALEdit.java:146)
at org.apache.hadoop.hbase.regionserver.wal.HLog$Entry.readFields(HLog.java:1767)
at org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682)
... 11 more

Issue Links

• is duplicated by: HBASE-6534 [replication] replication will be block if WAL compress set differently in master and slave configuration
• is duplicated by: HBASE-6535 [replication] replication will be block if WAL compress set differently in master and slave cluster configuration
• is duplicated by: HBASE-6536 [replication] replication will be block if WAL compress set differently in master and slave cluster configuration
• is related to: HBASE-5778 Fix HLog compression's incompatibilities