[IMPALA-6044] Analytic planner misses some opportunities to merge partition groups

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Impala 2.7.0
• Component/s: Frontend
• Target Version: Product Backlog
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: Zhangyi Lu
• Created: 12/Oct/17 15:33
• Updated: 22/Nov/17 19:46

Description

Version: Impala Shell v2.7.0-cdh5.9.1 (24ad6df) built on Wed Jan 11 13:39:25 PST 2017

I used analytic function sum with over clause as below,

sum(power(sedol_weight_in_collective_all, 2) / manager_cnt_per_sedol * power(residual_risk, 2) * 12) over (partition by activity_date_id) as total_stock_specific_risk

Even there was no explicit order by clause in over clause, query profile(see attached screenshot) shows impala sorted records according to partition keys by default. I think for sum function, records within subgroups should not be sorted. There is not any help for sorting instead, it downgrades performance.

Issue Links

• is related to: IMPALA-6226 Analytic planner misses some opportunities to merge sort groups
• relates to: IMPALA-3357 Sorter's quicksort implementation is very suboptimal for duplicate keys