When we shut down a Drillbit, we should provide some options similar to those available from Oracle's shutdown command (see https://docs.oracle.com/cd/B28359_01/server.111/b28310/start003.htm#ADMIN11156) .
At present, in order to avoid problems like DRILL-2654, we try to do a short wait for executing queries, but that times out after 5 seconds, and doesn't help with long-running queries.
Someone that is running a long query might be unhappy about losing work for something that was near completion, so we can do better.
And, in order to avoid spurious cleanup problems and exceptions, we should explicitly cancel any remaining queries before we do complete the shutdown.
As in the Oracle example, we might have shutdown immediate issue cancellations to the running queries. A clean shutdown might not have a timeout, or might allow the specification of a longer timeout, and even when the timeout goes off, we should still cleanly cancel any remaining queries, and wait for the cancellations to complete.