Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Later
-
None
-
None
-
ghx-label-5
Description
Environment description: cluster scale (50/100/150 nodes and terabyte of ram available) - Admission Control enabled.
Issue description:
Despite the coordinator chosen (with data and statistics unchanged) a query will be planned always in the same way based on the metainfo that the coordinator have.
The query will be scheduled always on the same nodes if the memory requirements for the admission are satisfied:
https://github.com/cloudera/Impala/blob/cdh5-2.7.0_5.9.1/be/src/scheduling/admission-controller.cc#L307-L333
Equal queries are planned/scheduled always in the same way (to hit always the same nodes).
This often lead to queue the queries that are hitting the same nodes are queued (not admitted) as on those nodes there's no more memory available within its process limit despite the pool have lot of free memory and the overall cluster load is low.
When the plan is finished and the query can be evaluated to be admitted often happen that the admission is denied because one of the node have not enough memory to run the query operation (and the query is moved in the pool queue) despite the cluster have 50/100/150 nodes and terabyte of ram available.
Why the scheduler does not take in consideration the memory available on the nodes involved in the query before to buid the schedule, (maybe preferring a remote read/operation on a free memory node instead to include in the plan always the same nodes that will end to be:
1- overloaded
2- the query will be not immediately admitted, risking to be timedout in the pool queue
Since 2.7 the REPLICA_PREFERENCE can possibly help, but it's not good enough as it does not prevent the scheduler to choose busy nodes (with the same potential effect: query queued for lack of resource on specific node despite there are terabytes of free memory).
Feature Request:
It would be good if Impala had an option to execute queries (even with worse performance) excluding the nodes overloaded and including different nodes in order to get the query immediately admitted and executed.