[PHOENIX-6687] The region server hosting the SYSTEM.CATALOG fails to serve any metadata requests as default handler pool threads are exhausted.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 5.1.0, 5.1.1, 4.16.1, 5.2.0, 5.1.2
• Fix Version/s: 5.2.0
• Component/s: core
• Assignee: Jacob Isaac
• Reporter: Jacob Isaac
• Created: 16/Apr/22 01:54
• Updated: 29/Aug/22 18:31
• Resolved: 29/Aug/22 18:31

Description

When SYSTEM.CATALOG region server is restarted and the server is experiencing heavy metadata call volume.

The stack traces indicate that all the default handler pool threads are waiting for the CQSI.init thread to finish initializing.
The CQSI.init thread itself cannot proceed since it cannot complete the second RPC call (org.apache.phoenix.query.ConnectionQueryServicesImpl.checkClientServerCompatibility) due to thread starvation.

For e.g
The following code turned the getTable(..) into needing an additional server-to-server RPC call when initializing a PhoenixConnection (CQSI.init) for the first time on the JVM. 
It is well-known that server-to-server RPC calls are prone to deadlocking due to thread pool exhaustion.

Issue Links

• links to: GitHub Pull Request #1440