[DRILL-2865] Drillbit runs out of memory on multiple consecutive CTAS

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Pending Closed
• Affects Version/s: 0.9.0
• Fix Version/s: 1.0.0
• Labels: no_verified_test
• Assignee: Steven Phillips
• Reporter: Victoria Markman
• Created: 23/Apr/15 21:34
• Updated: 10/Jul/15 22:13
• Resolved: 12/May/15 19:58

Description

Hardware configuration:

• single node
• 64GB RAM
  Drill configuration
  DRILL_MAX_DIRECT_MEMORY="8G"
  DRILL_MAX_HEAP="4G"
  `planner.enable_multiphase_agg` = false;
  `store.parquet.block-size` = 134217728;
  `planner.enable_mux_exchange` = false;
  `exec.min_hash_table_size` = 67108864;
  `planner.enable_hashagg` = true;
  `planner.width.max_per_node` = 23;

Aggregation query on TPCDS scale factor 1:

        select 
                ss_sold_date_sk , 
                ss_sold_time_sk , 
                ss_item_sk , 
                ss_customer_sk , 
                ss_cdemo_sk, 
                count(*) from store_sales
        group by 
                ss_sold_date_sk , 
                ss_sold_time_sk , 
                ss_item_sk , 
                ss_customer_sk , 
                ss_cdemo_sk
;

1. Executing CTAS with this query and store.format = 'parquet' fails on iteration #9 with this configuration consistently
2. Ran query by itself: 47 iterations successfully
3. Ran CTAS with this query and store.format = 'csv': - 30 iterations did not reproduce the problem

Attached:

• drillbit.log
• scripts.tar (contains script that reproduces OOM)