[IMPALA-11842] Improve memory estimation for streaming aggregate operator

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Epic Color: ghx-label-8
• Assignee: Riza Suminto
• Reporter: Abhishek Rawat
• Created: 12/Jan/23 19:01
• Updated: 14/Jul/23 09:20
• Resolved: 13/Jul/23 15:26

Description

Streaming aggregate operator can over estimate the peak memory and as a result Impala could request max memory allowed by admission controller. This impacts query concurrency and also causes unnecessary scaling.

Looking at some cases in the following profile snippet, the estimated peak memory (8.15 GB) is 20X from actual peak memory (416.07 MB).

    Estimated Per-Host Mem: 247067273376
    Request Pool: root.default
    Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB) impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB) impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB) impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00 MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34 GB) impala-executor-003-7:27010(1.35 GB)
    Per Host Number of Fragment Instances: impala-executor-003-5:27010(37) impala-executor-003-4:27010(38) impala-executor-003-6:27010(37) impala-executor-003-0:27010(38) impala-executor-003-8:27010(38) impala-executor-003-2:27010(37) impala-executor-003-1:27010(38) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1) impala-executor-003-3:27010(37) impala-executor-003-9:27010(37) impala-executor-003-7:27010(38)
    Latest admission queue reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
    Admission result: Admitted (queued)
    Initial admission queue reason: waited 83020 ms, reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
    Cluster Memory Admitted: 500.10 GB
    Executor Group: root.default-group-002
    ExecSummary: 
Operator                 #Hosts  #Inst   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                      
-----------------------------------------------------------------------------------------------------------------------------------------
F04:ROOT                      1      1  146.192us  146.192us                         4.01 MB        4.00 MB                              
11:MERGING-EXCHANGE           1      1    3.297ms    3.297ms      100         100    1.88 MB      234.53 KB  UNPARTITIONED               
F03:EXCHANGE SENDER          10    120    4.196ms  374.168ms                         7.52 KB              0                              
05:TOP-N                     10    120   22.184ms    1s028ms   12.00K         100   16.00 KB        1.56 KB                              
10:AGGREGATE                 10    120       4m2s     12m14s  499.98K     487.66K    2.34 MB       10.00 MB  FINALIZE                    
09:EXCHANGE                  10    120    5s336ms    7s799ms   22.11B     487.66K   10.40 MB        3.09 MB  HASH(vendor_id)             
F02:EXCHANGE SENDER          10    120   28s199ms   48s974ms                         4.16 MB              0                              
04:AGGREGATE                 10    120      1m17s      1m34s   22.11B     487.66K   21.02 MB       10.00 MB  STREAMING                   
08:AGGREGATE                 10    120     12m29s     22m36s   50.00B      50.00B    3.85 GB       10.87 GB                              
07:EXCHANGE                  10    120   10s165ms   12s246ms   50.00B      50.00B   11.69 MB       12.34 MB  HASH(vendor_id,purchase_id) 
F00:EXCHANGE SENDER          10    120      1m28s      1m49s                         4.16 MB              0                              
03:AGGREGATE                 10    120      2m34s       5m5s   50.00B      50.00B  416.07 MB        8.15 GB  STREAMING                   
02:HASH JOIN                 10    120      1m17s      1m40s   50.00B      50.00B   46.12 KB              0  INNER JOIN, BROADCAST       
|--F05:JOIN BUILD            10     10  557.800ms  613.307ms                       408.02 MB      408.00 MB                              
|  06:EXCHANGE               10     10   88.770ms  102.048ms    5.00M       5.00M   16.11 MB       10.10 MB  BROADCAST                   
|  F01:EXCHANGE SENDER        5      5  190.673ms  212.904ms                        75.23 KB              0                              
|  00:SCAN HDFS               5      5  774.428ms  965.865ms    5.00M       5.00M   12.92 MB       64.00 MB  tab.product               
01:SCAN HDFS                 10    120      5m48s     13m14s   50.00B      50.00B   32.92 MB       88.00 MB  tab.pli 

Issue Links

• depends upon: IMPALA-12183 Maintain cardinality clamping across multi-phase aggregation