Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-2865

Drillbit runs out of memory on multiple consecutive CTAS

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Pending Closed
    • 0.9.0
    • 1.0.0
    • None

    Description

      Hardware configuration:

      • single node
      • 64GB RAM
        Drill configuration
        DRILL_MAX_DIRECT_MEMORY="8G"
        DRILL_MAX_HEAP="4G"
        `planner.enable_multiphase_agg` = false;
        `store.parquet.block-size` = 134217728;
        `planner.enable_mux_exchange` = false;
        `exec.min_hash_table_size` = 67108864;
        `planner.enable_hashagg` = true;
        `planner.width.max_per_node` = 23;

      Aggregation query on TPCDS scale factor 1:

              select 
                      ss_sold_date_sk , 
                      ss_sold_time_sk , 
                      ss_item_sk , 
                      ss_customer_sk , 
                      ss_cdemo_sk, 
                      count(*) from store_sales
              group by 
                      ss_sold_date_sk , 
                      ss_sold_time_sk , 
                      ss_item_sk , 
                      ss_customer_sk , 
                      ss_cdemo_sk
      ;
      

      1. Executing CTAS with this query and store.format = 'parquet' fails on iteration #9 with this configuration consistently
      2. Ran query by itself: 47 iterations successfully
      3. Ran CTAS with this query and store.format = 'csv': - 30 iterations did not reproduce the problem

      Attached:

      • drillbit.log
      • scripts.tar (contains script that reproduces OOM)

      Attachments

        1. drill-2865-no-compression.log
          1.01 MB
          Victoria Markman
        2. drillbit.log
          228 kB
          Victoria Markman
        3. scritpts.tar
          3 kB
          Victoria Markman

        Activity

          People

            sphillips Steven Phillips
            vicky Victoria Markman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: