[IMPALA-9814] Analytic planner can under-parallelise with mt_dop

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Frontend
• Labels: multithreading
• Epic Color: ghx-label-7
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 01/Jun/20 22:48
• Updated: 11/Jun/20 18:07
• Resolved: 11/Jun/20 18:07

Description

This is a TODO-MT in AnalyticPlanner.java. Here is an example of a query that is under-parallelised

    
    set mt_dop=3;
    explain select count(*) over (partition by ss_addr_sk, ss_store_sk), count(*) over (partition by ss_sold_date_sk, ss_st
ore_sk)
    from tpcds_parquet.store_sales;
    
    +---------------------------------------------------------------------------+
    | Explain String                                                            |
    +---------------------------------------------------------------------------+
    | Max Per-Host Resource Reservation: Memory=61.50MB Threads=7               |
    | Per-Host Resource Estimates: Memory=148MB                                 |
    |                                                                           |
    | PLAN-ROOT SINK                                                            |
    | |                                                                         |
    | 06:EXCHANGE [UNPARTITIONED]                                               |
    | |                                                                         |
    | 04:ANALYTIC                                                               |
    | |  functions: count(*)                                                    |
    | |  partition by: ss_sold_date_sk, ss_store_sk                             |
    | |  row-size=28B cardinality=2.88M                                         |
    | |                                                                         |
    | 03:SORT                                                                   |
    | |  order by: ss_sold_date_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST |
    | |  row-size=20B cardinality=2.88M                                         |
    | |                                                                         |
    | 02:ANALYTIC                                                               |
    | |  functions: count(*)                                                    |
    | |  partition by: ss_addr_sk, ss_store_sk                                  |
    | |  row-size=20B cardinality=2.88M                                         |
    | |                                                                         |
    | 01:SORT                                                                   |
    | |  order by: ss_addr_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST      |
    | |  row-size=12B cardinality=2.88M                                         |
    | |                                                                         |
    | 05:EXCHANGE [HASH(ss_store_sk)]                                           |
    | |                                                                         |
    | 00:SCAN HDFS [tpcds_parquet.store_sales]                                  |
    |    HDFS partitions=1824/1824 files=1824 size=196.96MB                     |
    |    row-size=12B cardinality=2.88M                                         |
    +---------------------------------------------------------------------------+