[IMPALA-2378] Impalad exceeded its mem limit

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.3.0
• Labels: crash resource-management
• Environment: impalad version 2.3.0-cdh5-INTERNAL DEBUG (build 4d15c36ced13be9e5b147d708f74f6b4a869cb85); Built on Sat, 19 Sep 2015 02:08:23 PST
• Target Version: Impala 2.3.0
• Assignee: Tim Armstrong
• Reporter: casey
• Created: 20/Sep/15 00:10
• Updated: 20/Dec/15 00:05
• Resolved: 09/Oct/15 20:12

Description

During a stress test run, an impalad got oom killed. The oom killer says it was using a little over 14gb of memory though impalad is only configured to use a little less than 12.

dmesg

[ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
[  324]     0   324     2667      122   0     -17         -1000 udevd
[  793]     0   793     2280      135   1       0             0 dhclient
[  838]     0   838     1540       31   1       0             0 portreserve
[  846]     0   846    63872      161   0       0             0 rsyslogd
[  875]    81   875     5358       80   0       0             0 dbus-daemon
[  912]    68   912     9455      171   0       0             0 hald
[  913]     0   913     5100       68   1       0             0 hald-runner
[  973]    28   973   200417      151   3       0             0 nscd
[ 1200]     0  1200    19901      241   2       0             0 master
[ 1223]    89  1223    19910      236   2       0             0 qmgr
[ 1226]     0  1226    28187       53   1       0             0 abrtd
[ 1236]     0  1236    28741      160   2       0             0 crond
[ 1255]     0  1255     4856       46   1       0             0 atd
[ 1426]     0  1426     1020       31   0       0             0 agetty
[ 1660]     0  1660    16081      185   0     -17         -1000 sshd
[ 4670]    38  4670     6313      161   0       0             0 ntpd
[ 5395]     0  5395    26577       71   0       0             0 mysqld_safe
[ 5518]    27  5518   107922     6197   0       0             0 mysqld
[ 6016]     0  6016    35776      102   0       0             0 su
[ 6018]     0  6018   566865    11922   0       0             0 python
[ 6054]     0  6054    50142     2726   0       0             0 python
[ 6055]     0  6055    48444     2181   0       0             0 python
[ 8141]   495  8141   292124   117084   1       0             0 java
[ 8337]   494  8337   295668    72946   3       0             0 java
[ 4968]   482  4968 12946355  3545992   0       0             0 impalad
[29174]    89 29174    19868      233   2       0             0 pickup
Out of memory: Kill process 4968 (impalad) score 925 or sacrifice child
Killed process 4968, UID 482, (impalad) total-vm:51785420kB, anon-rss:14171580kB, file-rss:12388kB

The last message in the impalad log is

I0919 16:43:21.456208  9115 buffered-block-mgr.cc:399] Query: ae478af2729ddc96:3fd0ed078556a38c. Node=9 ran out of memory:
Buffered block mgr
  Num writes outstanding: 0
  Num free io buffers: 0
  Num unpinned blocks: 0
  Num available buffers: -324
  Total pinned buffers: 49
  Unfullfilled reserved buffers: 203
  Remaining memory: -1017198804 (#blocks=-121)
  Block write threshold: 4
Client 0x1971ec750
  num_reserved_buffers=34
  num_tmp_reserved_buffers=0
  num_pinned_buffers=0

a few seconds earlier there was a message that showed the process mem limit was exceeded by a little

I0919 16:43:18.965378  8958 runtime-state.cc:228] Error from query 7148aa15d9c6aaa2:388b03fa4a28686: Memory Limit Exceeded
Process: memory limit exceeded. Limit=11.73 GB Consumption=11.91 GB

In CM the impalad is configured to use 12592155852 bytes.

Issue Links

• is blocked by: IMPALA-2380 Nested Loop Join should check mem limit when deep copying
• is blocked by: IMPALA-2381 Row batches should count their tuple pointers towards memory limit