[HBASE-17200] Document an interesting implication of HBASE-15212

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Component/s: documentation, Operability, Replication
• Assignee: Unassigned
• Reporter: Andrew Kyle Purtell
• Created: 29/Nov/16 23:17
• Updated: 11/Jun/22 23:23
• Resolved: 11/Jun/22 23:23

Description

We had a Phoenix client application unfortunately batching up 1000 rows at a time. Phoenix bundles mutations up only considering the row count not byte count (see PHOENIX-541) so this lead to some single WALEdits in excess of 600 MB. A cluster without max RPC size enforcement accepted them. (That may be something we should fix - WALEdits that large are crazy.) Then replication workers attempting to ship the monster edits from this cluster to a remote cluster recently upgraded with RPC size enforcement active would see all their RPC attempts rejected, because the default limit is 256 MB.

This is an edge case but I can see it happening in practice and taking users by surprise, most likely when replicating between mixed versions. We should document this in the troubleshooting section.

Issue Links

• relates to: HBASE-16752 Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
• relates to: HBASE-15212 RPCServer should enforce max request size