Details
-
Sub-task
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
1.3.0
-
None
-
None
Description
Current queue support relies on a distributed semaphore around a fix pre-defined number. This semaphore indicates the number of queries Drill can run concurrently. Presently, we define small and large queues where we classify queries based on a threshold and use two semaphores around small and large queues individually.
This issue proposes to come up with an enhanced queueing or query dispatch mechanism where a query is granted execution based on its cost and availability of system resources(cpu, io, memory etc). Enhancing cost planing and introducing a distributed resource management should be addressed later to fully benefit from this enhancement. The proposal is a non-blocking and asynchronous mechanism that assumes eventual consistency around available system resources.