[IMPALA-5161] TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.9.0
• Target Version: Impala 2.9.0
• Epic Color: ghx-label-5
• Assignee: Tim Armstrong
• Reporter: Mostafa Mokhtar
• Created: 04/Apr/17 18:27
• Updated: 17/Apr/17 18:03
• Resolved: 17/Apr/17 18:03

Description

I set the memory limit to 10GB which is fairly low for Q78 and it failed with:

Query (id=8d463de3722df36f:a2dae93a00000000):
  Summary:
    Session ID: b54cda644686da93:f1c20afc188e0fa0
    Session Type: BEESWAX
    Start Time: 2017-04-04 10:46:17.099246000
    End Time: 2017-04-04 11:06:04.438426000
    Query Type: QUERY
    Query State: EXCEPTION
    Query Status: 
Cannot perform aggregation at node with id 30. Repartitioning did not reduce the size of a spilled partition. Repartitioning level 7. Number of rows 1.

    Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (build 07021cc117c56e4bbc0744b342fee6f5a8cce964)
    User: mmokhtar
    Connected User: mmokhtar
    Delegated User: 
    Network Address: 10.17.234.19:45450
    Default Db: tpcds_10000_parquet
    Sql Statement: with ws as (select d_year AS ws_sold_year, ws_item_sk, ws_bill_customer_sk ws_customer_sk, sum(ws_quantity) ws_qty, sum(ws_wholesale_cost) ws_wc, sum(ws_sales_price) ws_sp from web_sales left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk join date_dim on ws_sold_date_sk = d_date_sk where wr_order_number is null group by d_year, ws_item_sk, ws_bill_customer_sk ), cs as (select d_year AS cs_sold_year, cs_item_sk, cs_bill_customer_sk cs_customer_sk, sum(cs_quantity) cs_qty, sum(cs_wholesale_cost) cs_wc, sum(cs_sales_price) cs_sp from catalog_sales left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk join date_dim on cs_sold_date_sk = d_date_sk where cr_order_number is null group by d_year, cs_item_sk, cs_bill_customer_sk ), ss as (select d_year AS ss_sold_year, ss_item_sk, ss_customer_sk, sum(ss_quantity) ss_qty, sum(ss_wholesale_cost) ss_wc, sum(ss_sales_price) ss_sp from store_sales left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk join date_dim on ss_sold_date_sk = d_date_sk where sr_ticket_number is null group by d_year, ss_item_sk, ss_customer_sk ) select ss_item_sk, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio, ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price, coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty, coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost, coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price from ss left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk) left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and cs_customer_sk=ss_customer_sk) where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000 order by ss_item_sk, ss_qty desc, ss_wc desc, ss_sp desc, other_chan_qty, other_chan_wholesale_cost, other_chan_sales_price, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) limit 100
    Coordinator: vd1309.halxg.cloudera.com:22000
    Query Options (non default): MEM_LIMIT=10737418240

Issue Links

• duplicates: IMPALA-2708 Partitioned aggregation node repartitions when spilled partition could fit in memory
• relates to: IMPALA-3200 Replace BufferedBlockMgr with new buffer pool