Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3070

Memory Leak when we run out of memory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • Execution - Flow
    • None

    Description

      git.commit.id.abbrev=d10769f

      No of Drillbits : 2
      DRILL_MAX_DIRECT_MEMORY="1G"

      I ran the below query on tpsh SF 100

      select
        l.l_shipmode,
        sum(case
          when o.o_orderpriority = '1-URGENT'
            or o.o_orderpriority = '2-HIGH'
            then 1
          else 0
        end) as high_line_count,
        sum(case
          when o.o_orderpriority <> '1-URGENT'
            and o.o_orderpriority <> '2-HIGH'
            then 1
          else 0
        end) as low_line_count
      from
        orders o,
        lineitem l
      where
        o.o_orderkey = l.l_orderkey
        and l.l_shipmode in ('TRUCK', 'REG AIR')
        and l.l_commitdate < l.l_receiptdate
        and l.l_shipdate < l.l_commitdate
        and l.l_receiptdate >= date '1994-01-01'
        and l.l_receiptdate < date '1994-01-01' + interval '1' year
      group by
        l.l_shipmode
      order by
        l.l_shipmode;
      +------------+-----------------+----------------+
      | l_shipmode | high_line_count | low_line_count |
      +------------+-----------------+----------------+
      java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: org.apache.drill.exec.memory.OutOfMemoryRuntimeException: Failure while reallocating buffer to 262152 bytes
      
      Fragment 7:30
      
      [Error Id: 65a09d7d-75f2-4bbb-b8f9-cfb0bdfc2655 on qa-node191.qa.lab:31010]
      	at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
      	at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
      	at sqlline.SqlLine.print(SqlLine.java:1809)
      	at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
      	at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
      	at sqlline.SqlLine.dispatch(SqlLine.java:889)
      	at sqlline.SqlLine.begin(SqlLine.java:763)
      	at sqlline.SqlLine.start(SqlLine.java:498)
      	at sqlline.SqlLine.main(SqlLine.java:460)
      

      Before memory :

      0: jdbc:drill:schema=dfs_eea> select * from sys.memory;
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      |  hostname  | user_port  | heap_current |  heap_max  | direct_current | jvm_direct_current | direct_max |
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      | qa-node190.qa.lab | 31010      | 126726408    | 4151836672 | 11799087       | 100676940          | 1073741824 |
      | qa-node191.qa.lab | 31010      | 299617984    | 4151836672 | 7750365        | 50345292           | 1073741824 |
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      

      After Memory :

      0: jdbc:drill:schema=dfs_eea> select * from sys.memory;
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      |  hostname  | user_port  | heap_current |  heap_max  | direct_current | jvm_direct_current | direct_max |
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      | qa-node190.qa.lab | 31010      | 446765888    | 4151836672 | 161309493      | 436221380          | 1073741824 |
      | qa-node191.qa.lab | 31010      | 357580248    | 4151836672 | 199260917      | 603993536          | 1073741824 |
      +------------+------------+--------------+------------+----------------+--------------------+------------+
      

      While I expected this OOM error (as direct mem is 1GB), there seems to be a bump in directory_memory on qa-node191.qa.lab. Not sure if this is normal

      • Rahul

      Attachments

        1. node1.log
          4.74 MB
          Rahul Kumar Challapalli
        2. node2.log
          3.57 MB
          Rahul Kumar Challapalli

        Activity

          People

            jaltekruse Jason Altekruse
            rkins Rahul Kumar Challapalli
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: