Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2378

Impalad exceeded its mem limit

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.3.0
    • Impala 2.3.0
    • None
    • impalad version 2.3.0-cdh5-INTERNAL DEBUG (build 4d15c36ced13be9e5b147d708f74f6b4a869cb85)
      Built on Sat, 19 Sep 2015 02:08:23 PST

    Description

      During a stress test run, an impalad got oom killed. The oom killer says it was using a little over 14gb of memory though impalad is only configured to use a little less than 12.

      dmesg

      [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
      [  324]     0   324     2667      122   0     -17         -1000 udevd
      [  793]     0   793     2280      135   1       0             0 dhclient
      [  838]     0   838     1540       31   1       0             0 portreserve
      [  846]     0   846    63872      161   0       0             0 rsyslogd
      [  875]    81   875     5358       80   0       0             0 dbus-daemon
      [  912]    68   912     9455      171   0       0             0 hald
      [  913]     0   913     5100       68   1       0             0 hald-runner
      [  973]    28   973   200417      151   3       0             0 nscd
      [ 1200]     0  1200    19901      241   2       0             0 master
      [ 1223]    89  1223    19910      236   2       0             0 qmgr
      [ 1226]     0  1226    28187       53   1       0             0 abrtd
      [ 1236]     0  1236    28741      160   2       0             0 crond
      [ 1255]     0  1255     4856       46   1       0             0 atd
      [ 1426]     0  1426     1020       31   0       0             0 agetty
      [ 1660]     0  1660    16081      185   0     -17         -1000 sshd
      [ 4670]    38  4670     6313      161   0       0             0 ntpd
      [ 5395]     0  5395    26577       71   0       0             0 mysqld_safe
      [ 5518]    27  5518   107922     6197   0       0             0 mysqld
      [ 6016]     0  6016    35776      102   0       0             0 su
      [ 6018]     0  6018   566865    11922   0       0             0 python
      [ 6054]     0  6054    50142     2726   0       0             0 python
      [ 6055]     0  6055    48444     2181   0       0             0 python
      [ 8141]   495  8141   292124   117084   1       0             0 java
      [ 8337]   494  8337   295668    72946   3       0             0 java
      [ 4968]   482  4968 12946355  3545992   0       0             0 impalad
      [29174]    89 29174    19868      233   2       0             0 pickup
      Out of memory: Kill process 4968 (impalad) score 925 or sacrifice child
      Killed process 4968, UID 482, (impalad) total-vm:51785420kB, anon-rss:14171580kB, file-rss:12388kB
      

      The last message in the impalad log is

      I0919 16:43:21.456208  9115 buffered-block-mgr.cc:399] Query: ae478af2729ddc96:3fd0ed078556a38c. Node=9 ran out of memory:
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -324
        Total pinned buffers: 49
        Unfullfilled reserved buffers: 203
        Remaining memory: -1017198804 (#blocks=-121)
        Block write threshold: 4
      Client 0x1971ec750
        num_reserved_buffers=34
        num_tmp_reserved_buffers=0
        num_pinned_buffers=0
      

      a few seconds earlier there was a message that showed the process mem limit was exceeded by a little

      I0919 16:43:18.965378  8958 runtime-state.cc:228] Error from query 7148aa15d9c6aaa2:388b03fa4a28686: Memory Limit Exceeded
      Process: memory limit exceeded. Limit=11.73 GB Consumption=11.91 GB
      

      In CM the impalad is configured to use 12592155852 bytes.

      Attachments

        1. heap-tpcds-7.pdf
          16 kB
          Tim Armstrong
        2. IMPALA-2378-pool-dump.txt
          29 kB
          Tim Armstrong

        Issue Links

          Activity

            People

              tarmstrong Tim Armstrong
              caseyc casey
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: