[HBASE-13480] ShortCircuitConnection doesn't short-circuit all calls as expected

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 1.1.0, 2.0.0
• Fix Version/s: 1.0.2, 1.2.0, 1.3.0, 1.1.3, 2.0.0
• Component/s: Client
• Hadoop Flags: Reviewed
• Assignee: Jingcheng Du
• Reporter: Josh Elser
• Created: 15/Apr/15 20:01
• Updated: 27/Jan/16 15:02
• Resolved: 24/Aug/15 19:39

Description

Noticed the following situation in debugging unexpected unit tests failures in HBASE-13351.

ConnectionUtils#createShortCircuitHConnection(Connection, ServerName, AdminService.BlockingInterface, ClientService.BlockingInterface) is intended to avoid the extra RPC by calling the server's instantiation of the protobuf rpc stub directly for the AdminService and ClientService.

The problem is that this is insufficient to actually avoid extra "remote" RPCs as all other calls to the Connection are routed to a "real" Connection instance. As such, any object created by the "real" Connection (such as an HTable) will use the real Connection, not the SSC.

The end result is that MasterRpcService#reportRegionStateTransition(RpcController, ReportRegionStateTransitionRequest) will make additional "remote" RPCs over what it thinks is an SSC through a Get on HTable which was constructed using the SSC, but the Get itself will use the underlying real Connection instead of the SSC. With insufficiently sized thread pools, this has been observed to result in RPC deadlock in the HMaster where an RPC attempts to make another RPC but there are no more threads available to service the second RPC so the first RPC blocks indefinitely.

Issue Links

• is related to: HBASE-14290 Spin up less threads in tests