Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.3.0
-
None
-
impalad version 2.3.0-cdh5-INTERNAL DEBUG (build 4d15c36ced13be9e5b147d708f74f6b4a869cb85)
Built on Sat, 19 Sep 2015 02:08:23 PST
Description
During a stress test run, an impalad got oom killed. The oom killer says it was using a little over 14gb of memory though impalad is only configured to use a little less than 12.
dmesg
[ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name [ 324] 0 324 2667 122 0 -17 -1000 udevd [ 793] 0 793 2280 135 1 0 0 dhclient [ 838] 0 838 1540 31 1 0 0 portreserve [ 846] 0 846 63872 161 0 0 0 rsyslogd [ 875] 81 875 5358 80 0 0 0 dbus-daemon [ 912] 68 912 9455 171 0 0 0 hald [ 913] 0 913 5100 68 1 0 0 hald-runner [ 973] 28 973 200417 151 3 0 0 nscd [ 1200] 0 1200 19901 241 2 0 0 master [ 1223] 89 1223 19910 236 2 0 0 qmgr [ 1226] 0 1226 28187 53 1 0 0 abrtd [ 1236] 0 1236 28741 160 2 0 0 crond [ 1255] 0 1255 4856 46 1 0 0 atd [ 1426] 0 1426 1020 31 0 0 0 agetty [ 1660] 0 1660 16081 185 0 -17 -1000 sshd [ 4670] 38 4670 6313 161 0 0 0 ntpd [ 5395] 0 5395 26577 71 0 0 0 mysqld_safe [ 5518] 27 5518 107922 6197 0 0 0 mysqld [ 6016] 0 6016 35776 102 0 0 0 su [ 6018] 0 6018 566865 11922 0 0 0 python [ 6054] 0 6054 50142 2726 0 0 0 python [ 6055] 0 6055 48444 2181 0 0 0 python [ 8141] 495 8141 292124 117084 1 0 0 java [ 8337] 494 8337 295668 72946 3 0 0 java [ 4968] 482 4968 12946355 3545992 0 0 0 impalad [29174] 89 29174 19868 233 2 0 0 pickup Out of memory: Kill process 4968 (impalad) score 925 or sacrifice child Killed process 4968, UID 482, (impalad) total-vm:51785420kB, anon-rss:14171580kB, file-rss:12388kB
The last message in the impalad log is
I0919 16:43:21.456208 9115 buffered-block-mgr.cc:399] Query: ae478af2729ddc96:3fd0ed078556a38c. Node=9 ran out of memory: Buffered block mgr Num writes outstanding: 0 Num free io buffers: 0 Num unpinned blocks: 0 Num available buffers: -324 Total pinned buffers: 49 Unfullfilled reserved buffers: 203 Remaining memory: -1017198804 (#blocks=-121) Block write threshold: 4 Client 0x1971ec750 num_reserved_buffers=34 num_tmp_reserved_buffers=0 num_pinned_buffers=0
a few seconds earlier there was a message that showed the process mem limit was exceeded by a little
I0919 16:43:18.965378 8958 runtime-state.cc:228] Error from query 7148aa15d9c6aaa2:388b03fa4a28686: Memory Limit Exceeded Process: memory limit exceeded. Limit=11.73 GB Consumption=11.91 GB
In CM the impalad is configured to use 12592155852 bytes.
Attachments
Attachments
Issue Links
- is blocked by
-
IMPALA-2380 Nested Loop Join should check mem limit when deep copying
- Resolved
-
IMPALA-2381 Row batches should count their tuple pointers towards memory limit
- Resolved