[HBASE-16752] Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0, 2.0.0
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Replication, rpc
• Hadoop Flags: Reviewed
• Assignee: Ashu Pachauri
• Reporter: Ashu Pachauri
• Created: 03/Oct/16 19:56
• Updated: 09/Nov/17 02:49
• Resolved: 19/Oct/16 23:04

Description

In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by default to 256 MB. This means that during upgrade scenarios (or when source is 1.2 peer is already on 1.3), it's possible to encounter a situation where we try to send an rpc with size greater than 256 MB because we never unroll a WALEdit while sending replication traffic.

RpcServer throws the underlying exception locally, but closes the connection with returning the underlying error to the client, and client only sees a "Broken pipe" error.
I am not sure what is the proper fix here (or if one is needed) to make sure this does not happen, but we should return the underlying exception to the RpcClient, because without it, it can be difficult to diagnose the problem, especially for someone new to HBase.

Issue Links

• is related to: HBASE-17200 Document an interesting implication of HBASE-15212
• relates to: HBASE-15212 RPCServer should enforce max request size