[PHOENIX-2500] Small race condition for clients to see application-level exception when PQS shutting down

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Josh Elser
• Reporter: Josh Elser
• Created: 09/Dec/15 01:28
• Updated: 09/Dec/15 01:28

Description

I wrote a little test for PQS where multiple instances sit behind a dumb load balancer. A client thread runs queries against the load balancer, and another thread willy-nilly kills PQS instances (chaos-monkey esque).

We noticed that when a PQS instance gets killed while a query is running, a client might get an exception stemming from the ThreadPool being shut down by the PhoenixDriver ShutdownHook.

Ideally, the client would just see an HTTP/500-level error or get some sort of transport-level exception, not an exception with an RejectedExecutionException. The solution, I think, is to let PQS register its own code to run before PhoenixDriver closes itself.