[HBASE-4246] Cluster with too many regions cannot withstand some master failover scenarios

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.90.4
• Component/s: master, Zookeeper
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 23/Aug/11 23:22
• Updated: 12/Jun/22 19:18
• Resolved: 16/Nov/16 20:42

Description

We ran into the following sequence of events:

• master startup failed after only ROOT had been assigned (for another reason)
• restarted the master without restarting other servers. Since there was at least one region assigned, it went through the failover code path
• master scanned META and inserted every region into /hbase/unassigned in ZK.
• then, it called "listChildren" on the /hbase/unassigned znode, and crashed with "Packet len6080218 is out of range!" since the IPC response was larger than the default maximum.

Issue Links

• is broken by: ZOOKEEPER-1162 consistent handling of jute.maxbuffer when attempting to read large zk "directories"
• is related to: HBASE-24544 Recommend upping zk jute.maxbuffer in all but minor installs
• is related to: HBASE-6625 If we have hundreds of thousands of regions getChildren will encouter zk exception