[IMPALA-12988] Calculate an unbounded version of CpuAsk

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.4.0
• Component/s: Frontend
• Target Version: Impala 4.4.0
• Epic Color: ghx-label-7
• Assignee: Riza Suminto
• Reporter: Riza Suminto
• Created: 09/Apr/24 18:58
• Updated: 22/Apr/24 14:21
• Resolved: 20/Apr/24 14:14

Description

CpuAsk is calculated through recursive call beginning at Planner.computeBlockingAwareCores(), which called after Planner.computeEffectiveParallelism(). It does blocking operator analysis over selected degree of parallelism that decided during computeEffectiveParallelism() traversal. That selected degree of parallelism, however, is already bounded by min and max parallelism config, derived from PROCESSING_COST_MIN_THREADS and MAX_FRAGMENT_INSTANCES_PER_NODE options accordingly.

It is beneficial to have another version of CpuAsk that is not bounded by min and max parallelism config. It should purely based on the fragment's ProcessingCost and query plan relationship constraint (ie., num JOIN BUILDER fragment should be equal as num JOIN fragment for partitioned join). During executor group set selection, Frontend should use the unbounded CpuAsk number to avoid assigning query to small executor group set prematurely.