Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9814

Analytic planner can under-parallelise with mt_dop

    XMLWordPrintableJSON

    Details

    • Epic Color:
      ghx-label-7

      Description

      This is a TODO-MT in AnalyticPlanner.java. Here is an example of a query that is under-parallelised

          
          set mt_dop=3;
          explain select count(*) over (partition by ss_addr_sk, ss_store_sk), count(*) over (partition by ss_sold_date_sk, ss_st
      ore_sk)
          from tpcds_parquet.store_sales;
          
          +---------------------------------------------------------------------------+
          | Explain String                                                            |
          +---------------------------------------------------------------------------+
          | Max Per-Host Resource Reservation: Memory=61.50MB Threads=7               |
          | Per-Host Resource Estimates: Memory=148MB                                 |
          |                                                                           |
          | PLAN-ROOT SINK                                                            |
          | |                                                                         |
          | 06:EXCHANGE [UNPARTITIONED]                                               |
          | |                                                                         |
          | 04:ANALYTIC                                                               |
          | |  functions: count(*)                                                    |
          | |  partition by: ss_sold_date_sk, ss_store_sk                             |
          | |  row-size=28B cardinality=2.88M                                         |
          | |                                                                         |
          | 03:SORT                                                                   |
          | |  order by: ss_sold_date_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST |
          | |  row-size=20B cardinality=2.88M                                         |
          | |                                                                         |
          | 02:ANALYTIC                                                               |
          | |  functions: count(*)                                                    |
          | |  partition by: ss_addr_sk, ss_store_sk                                  |
          | |  row-size=20B cardinality=2.88M                                         |
          | |                                                                         |
          | 01:SORT                                                                   |
          | |  order by: ss_addr_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST      |
          | |  row-size=12B cardinality=2.88M                                         |
          | |                                                                         |
          | 05:EXCHANGE [HASH(ss_store_sk)]                                           |
          | |                                                                         |
          | 00:SCAN HDFS [tpcds_parquet.store_sales]                                  |
          |    HDFS partitions=1824/1824 files=1824 size=196.96MB                     |
          |    row-size=12B cardinality=2.88M                                         |
          +---------------------------------------------------------------------------+
          
      
      

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: