Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3732

Drill leaks memory if external sort hits out of disk space exception

    XMLWordPrintableJSON

    Details

      Description

      Ran into it when running CTAS with partition by.

      Here is what reproduction looks like:

      0: jdbc:drill:schema=dfs> create table store_sales_4(ss_item_sk, ss_customer_sk, ss_cdemo_sk, ss_hdemo_sk, s_sold_date_sk, ss_promo_sk) partition by (ss_promo_sk) as
      . . . . . . . . . . . . > select
      . . . . . . . . . . . . >     case when columns[2] = '' then cast(null as varchar(100)) else cast(columns[2] as varchar(100)) end,
      . . . . . . . . . . . . >     case when columns[3] = '' then cast(null as varchar(100)) else cast(columns[3] as varchar(100)) end,
      . . . . . . . . . . . . >     case when columns[4] = '' then cast(null as varchar(100)) else cast(columns[4] as varchar(100)) end, 
      . . . . . . . . . . . . >     case when columns[5] = '' then cast(null as varchar(100)) else cast(columns[5] as varchar(100)) end, 
      . . . . . . . . . . . . >     case when columns[0] = '' then cast(null as varchar(100)) else cast(columns[0] as varchar(100)) end, 
      . . . . . . . . . . . . >     case when columns[8] = '' then cast(null as varchar(100)) else cast(columns[8] as varchar(100)) end
      . . . . . . . . . . . . > from
      . . . . . . . . . . . . >          `store_sales.dat` ss     
      . . . . . . . . . . . . > ;
      Error: SYSTEM ERROR: IllegalStateException: Failure while closing accountor.  Expected private and shared pools to be set to initial values.  However, one or more were not.  Stats are
      	zone	init	allocated	delta 
      	private	10000000	9680512	319488 
      	shared	10000000	10000000	0.
      
      Fragment 1:21
      
      [Error Id: bd0d7d59-8693-476b-8671-70f0b2e7a176 on atsqa4-133.qa.lab:31010] (state=,code=0)
      

      Setup:
      single node
      8GB direct memory
      4GB heap memory
      store_sales.dat is a file from TPCDS SF100

      drillbit.log attached

        Attachments

        1. drillbit.log
          9.58 MB
          Victoria Markman

          Activity

            People

            • Assignee:
              vicky Victoria Markman
              Reporter:
              vicky Victoria Markman
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: