Hello, I installed httpd 2.4.2 on CentOS6.2(64bit). It seems like httpd leaks memory when mod_ssl is used. When I start up httpd with mod_ssl, and when I issue "apachectl graceful", it seems like httpd leaks memory. If I issue "apachectl graceful" 1000 times, ps command shows the memory usage gets bigger like below: ====Before 1000 times graceful restart==== # ps -elfy | head -1 S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD # ps -elfy | grep httpd | grep -v grep S root 3816 1 0 80 0 4136 25683 poll_s 15:00 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 3818 3816 0 80 0 5216 111748 pipe_w 15:00 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 3819 3816 0 80 0 5220 111748 pipe_w 15:00 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 3820 3816 0 80 0 5220 111748 pipe_w 15:00 ? 00:00:00 /opt/testhttpd/bin/httpd # ====After 1000 times graceful restart==== # ps -elfy | head -1 S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD # ps -elfy | grep httpd | grep -v grep S root 3816 1 6 80 0 97320 48673 poll_s 15:00 ? 00:01:23 /opt/testhttpd/bin/httpd S daemon 28584 3816 0 80 0 97448 134738 pipe_w 15:19 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 28585 3816 0 80 0 95416 134738 pipe_w 15:19 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 28588 3816 0 80 0 97460 134738 pipe_w 15:19 ? 00:00:00 /opt/testhttpd/bin/httpd # You can reproduce this by: 1. start httpd with mod_ssl loaded. 2. issue "apachectl graceful" 1000 times. I configured my httpd with following: # ./configure --prefix=/opt/testhttpd --with-included-apr If I start up httpd without mod_ssl, memory usage does not get bigger even after 1000 times graceful restart. The evidence is below: ====Before 1000 times graceful restart==== # ps -elfy | head -1 S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD # ps -elfy | grep httpd | grep -v grep S root 25061 1 0 80 0 2632 18365 poll_s 15:43 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 25063 25061 0 80 0 4504 104430 pipe_w 15:43 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 25064 25061 0 80 0 4504 104430 pipe_w 15:43 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 25065 25061 0 80 0 4508 104430 pipe_w 15:43 ? 00:00:00 /opt/testhttpd/bin/httpd # ====After 1000 times graceful restart==== # ps -elfy | head -1 S UID PID PPID C PRI NI RSS SZ WCHAN STIME TTY TIME CMD # ps -elfy | grep httpd | grep -v grep S daemon 17566 25061 0 80 0 4784 104430 pipe_w 16:01 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 17567 25061 0 80 0 4788 104430 pipe_w 16:01 ? 00:00:00 /opt/testhttpd/bin/httpd S daemon 17568 25061 0 80 0 4792 104430 pipe_w 16:01 ? 00:00:00 /opt/testhttpd/bin/httpd S root 25061 1 0 80 0 3264 18365 poll_s 15:43 ? 00:00:06 /opt/testhttpd/bin/httpd # Are there memory leaks in mod_ssl module? Thanks, Tadanori
This was also brought up on httpd-dev in October 2008 (with no final conclusion, though): http://mail-archives.apache.org/mod_mbox/httpd-dev/200810.mbox/%3C48F91F01.4010802@force-elite.com%3E (For the sake of reference: a link to your original post to the users list - http://mail-archives.apache.org/mod_mbox/httpd-users/201206.mbox/%3C1339507051.6642.YahooMailClassic%40web140201.mail.bf1.yahoo.com%3E - would have been appreciated.)
Hi, i believe to have run into the same issue. Graceful restarts every 5 minutes fill up about 400mb of ram in a day. With mdb and ::findleaks i've tracked it down to stacktraces like this: ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 811e40 80fd20 40b3df2f559484 1 4bd668 0 0 libumem.so.1`umem_cache_alloc_debug+0xfd libumem.so.1`umem_cache_alloc+0xb3 libumem.so.1`umem_alloc+0x64 libumem.so.1`umem_malloc+0x3f libcrypto.so.1.0.0`CRYPTO_malloc+0x5d libcrypto.so.1.0.0`lh_insert+0x1fe libcrypto.so.1.0.0`int_err_set_item+0x4f libcrypto.so.1.0.0`ERR_load_strings+0x41 libssl.so.1.0.0`ERR_load_SSL_strings+0x2e mod_ssl.so`ssl_hook_pre_config+0x30 ap_run_pre_config+0x60 main+0x905 _start+0x6c Which sent me looking for ERR_free_strings(). It turns out that this was removed over 10 years ago in the following commit: https://github.com/apache/httpd/commit/e2ee089fd414675144e347fc4624b1a665aac671 I have build 2.4.10 with this change reverted and it absolutly fixes the issue for us. All tests were done with 2.4.10 on SmartOS (Illumos) from pkgsrc and openssl-1.0.1h. I'm not sure if the reason for the removal is still valid (maybe something changed in openssl). Neither am I sure that reverting that commit won't break the error strings (don't know how to test). Of course I'm also wondering why this is not a bigger problem for other users, but maybe they don't do graceful restarts that often ;-) So maybe someone who knows a bit more on that topic can help here...
Thanks Sebastian. I've committed the fix based on your debugging in r1638772.
Proposed to 2.4.x.
Backported to 2.4.11 in r1642404.