I'm runnig apache2.2.3, Win 2003 as reverse proxy , mod_rewrite; mod_proxy; mod_security The MaxRequestPerChild directive is set to 250, evertyhing is running smoothly, httpd.exe is recycling frequently After a certain load i an an error in the eventlog: Faulting application httpd.exe, version 2.2.3.0, faulting module libhttpd.dll, version 2.2.3.0, fault address 0x0001e440. From that time on httpd.exe is growing 10-20 Mbyte per Minute and i have to restart apache service. 1. Question what cause the error message and what cause the memory leak
"Faulting application httpd.exe, version 2.2.3.0, faulting module libhttpd.dll, version 2.2.3.0, fault address 0x0001e440." Need to see your dr. watson backtrace, please?
o.k here is error.log [Tue Jan 09 08:26:26 2007] [notice] mod_security/1.9.4 configured [Tue Jan 09 08:26:27 2007] [notice] Child 444: Child process is running [Tue Jan 09 08:26:27 2007] [notice] Child 444: Acquired the start mutex. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting 250 worker threads. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting thread to listen on port 443. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting thread to listen on port 80. [Tue Jan 09 08:29:25 2007] [notice] Child 3032: Terminating 1 threads that failed to exit.
[Tue Jan 09 08:26:26 2007] [notice] mod_security/1.9.4 configured [Tue Jan 09 08:26:27 2007] [notice] Child 444: Child process is running [Tue Jan 09 08:26:27 2007] [notice] Child 444: Acquired the start mutex. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting 250 worker threads. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting thread to listen on port 443. [Tue Jan 09 08:26:27 2007] [notice] Child 444: Starting thread to listen on port 80. [Tue Jan 09 08:29:25 2007] [notice] Child 3032: Terminating 1 threads that failed to exit.
Are there any known problems with hyperthreading an apache ? ErrorLog: [Tue Jan 09 20:50:53 2007] [notice] Child 4896: Process exiting because it reached MaxRequestsPerChild. Signaling the parent to restart a new child process. [Tue Jan 09 20:50:53 2007] [notice] Parent: Received restart signal -- Restarting the server. [Tue Jan 09 20:50:54 2007] [notice] Apache/2.2.3 (Win32) mod_ssl/2.2.3 OpenSSL/0.9.8d configured -- resuming normal operations [Tue Jan 09 20:50:54 2007] [notice] Server built: Sep 28 2006 21:57:41 [Tue Jan 09 20:50:54 2007] [notice] Parent: Created child process 4808 Warning: DocumentRoot [C:/www/docs/dummy-host.example.com] does not exist Warning: DocumentRoot [C:/www/docs/dummy-host2.example.com] does not exist [Tue Jan 09 20:50:54 2007] [notice] Child 4896: Released the start mutex [Tue Jan 09 20:50:55 2007] [notice] mod_security/1.9.4 configured [Tue Jan 09 20:50:55 2007] [notice] Child 4896: Waiting for 250 worker threads to exit. [Tue Jan 09 20:50:56 2007] [notice] Child 4808: Child process is running [Tue Jan 09 20:50:56 2007] [notice] Child 4808: Acquired the start mutex. [Tue Jan 09 20:50:56 2007] [notice] Child 4808: Starting 250 worker threads. [Tue Jan 09 20:50:56 2007] [notice] Child 4808: Starting thread to listen on port 443. [Tue Jan 09 20:50:56 2007] [notice] Child 4808: Starting thread to listen on port 80. [Tue Jan 09 20:53:55 2007] [notice] Child 4896: Terminating 1 threads that failed to exit. [Wed Jan 10 07:26:23 2007] [notice] Parent: Received restart signal -- Restarting the server. Warning: DocumentRoot [C:/www/docs/dummy-host.example.com] does not exist Warning: DocumentRoot [C:/www/docs/dummy-host2.example.com] does not exist [Wed Jan 10 07:26:23 2007] [notice] Child 4808: Exit event signaled. Child process is ending. [Wed Jan 10 07:26:24 2007] [notice] Apache/2.2.3 (Win32) mod_ssl/2.2.3 OpenSSL/0.9.8d configured -- resuming normal operations [Wed Jan 10 07:26:24 2007] [notice] Server built: Sep 28 2006 21:57:41 [Wed Jan 10 07:26:24 2007] [notice] Parent: Created child process 2764 Warning: DocumentRoot [C:/www/docs/dummy-host.example.com] does not exist Warning: DocumentRoot [C:/www/docs/dummy-host2.example.com] does not exist [Wed Jan 10 07:26:25 2007] [notice] mod_security/1.9.4 configured [Wed Jan 10 07:26:25 2007] [notice] Child 4808: Waiting for 249 worker threads to exit. [Wed Jan 10 07:26:25 2007] [crit] [Wed Jan 10 07:26:25 2007] file .\\server\\mpm\\winnt\\child.c, line 1120, assertion "(rv >= 0) && (rv < threads_created)" failed [Wed Jan 10 07:26:26 2007] [notice] Child 2764: Child process is running [Wed Jan 10 07:26:26 2007] [notice] Child 2764: Acquired the start mutex. [Wed Jan 10 07:26:26 2007] [notice] Child 2764: Starting 250 worker threads. [Wed Jan 10 07:26:26 2007] [notice] Child 2764: Starting thread to listen on port 443. [Wed Jan 10 07:26:26 2007] [notice] Child 2764: Starting thread to listen on port 80.
I met the same issue. anyone can help on this? [Fri May 18 01:32:41 2007] [notice] Disabled use of AcceptEx() WinSock2 API [Fri May 18 01:32:41 2007] [notice] Child 2928: Child process is running [Fri May 18 01:32:42 2007] [notice] Child 1324: Released the start mutex [Fri May 18 01:32:42 2007] [notice] Child 2928: Acquired the start mutex. [Fri May 18 01:32:42 2007] [notice] Child 2928: Starting 100 worker threads. [Fri May 18 01:32:42 2007] [notice] Child 1324: Waiting for 100 worker threads to exit. [Fri May 18 01:32:42 2007] [notice] Child 2928: Listening on port 80. [Fri May 18 01:32:46 2007] [notice] Child 1324: All worker threads have exited. [Fri May 18 01:32:46 2007] [notice] Child 1324: Child process is exiting [Fri May 18 01:36:50 2007] [notice] Child 2928: Process exiting because it reached MaxRequestsPerChild. Signaling the parent to restart a new child process. [Fri May 18 01:36:50 2007] [notice] Parent: Received restart signal -- Restarting the server. [Fri May 18 01:36:50 2007] [notice] Apache/2.2.4 (Win32) configured -- resuming normal operations [Fri May 18 01:36:50 2007] [notice] Server built: Jan 9 2007 23:17:20 [Fri May 18 01:36:50 2007] [notice] Parent: Created child process 4056 [Fri May 18 01:36:50 2007] [notice] Disabled use of AcceptEx() WinSock2 API [Fri May 18 01:36:50 2007] [notice] Child 4056: Child process is running [Fri May 18 01:36:51 2007] [notice] Child 2928: Released the start mutex [Fri May 18 01:36:51 2007] [notice] Child 4056: Acquired the start mutex. [Fri May 18 01:36:51 2007] [notice] Child 4056: Starting 100 worker threads. [Fri May 18 01:36:51 2007] [notice] Child 2928: Waiting for 100 worker threads to exit. [Fri May 18 01:36:51 2007] [notice] Child 4056: Listening on port 80. [Fri May 18 01:36:55 2007] [notice] Child 2928: All worker threads have exited. [Fri May 18 01:36:55 2007] [notice] Child 2928: Child process is exiting [Fri May 18 01:40:34 2007] [notice] Child 4056: Process exiting because it reached MaxRequestsPerChild. Signaling the parent to restart a new child process. [Fri May 18 01:40:34 2007] [notice] Parent: Received restart signal -- Restarting the server. [Fri May 18 01:40:34 2007] [notice] Apache/2.2.4 (Win32) configured -- resuming normal operations [Fri May 18 01:40:34 2007] [notice] Server built: Jan 9 2007 23:17:20 [Fri May 18 01:40:34 2007] [notice] Parent: Created child process 800 [Fri May 18 01:40:34 2007] [notice] Disabled use of AcceptEx() WinSock2 API [Fri May 18 01:40:34 2007] [notice] Child 800: Child process is running [Fri May 18 01:40:35 2007] [notice] Child 4056: Released the start mutex [Fri May 18 01:40:35 2007] [notice] Child 800: Acquired the start mutex. [Fri May 18 01:40:35 2007] [notice] Child 800: Starting 100 worker threads. [Fri May 18 01:40:35 2007] [notice] Child 4056: Waiting for 100 worker threads to exit. [Fri May 18 01:40:35 2007] [notice] Child 800: Listening on port 80. [Fri May 18 01:43:35 2007] [notice] Child 4056: Terminating 1 threads that failed to exit. Event Type: Error Event Source: Application Error Event Category: (100) Event ID: 1000 Date: 2007-5-18 Time: 1:43:39 User: N/A Computer: PIANYIBA Description: Faulting application httpd.exe, version 2.2.4.0, faulting module libhttpd.dll, version 2.2.4.0, fault address 0x0001c9ca. For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp. Data: 0000: 41 70 70 6c 69 63 61 74 Applicat 0008: 69 6f 6e 20 46 61 69 6c ion Fail 0010: 75 72 65 20 20 68 74 74 ure htt 0018: 70 64 2e 65 78 65 20 32 pd.exe 2 0020: 2e 32 2e 34 2e 30 20 69 .2.4.0 i 0028: 6e 20 6c 69 62 68 74 74 n libhtt 0030: 70 64 2e 64 6c 6c 20 32 pd.dll 2 0038: 2e 32 2e 34 2e 30 20 61 .2.4.0 a 0040: 74 20 6f 66 66 73 65 74 t offset 0048: 20 30 30 30 31 63 39 63 0001c9c 0050: 61 a
reassign...
The -backtrace- I refer to is spelled out at http://httpd.apache.org/dev/debugging.html#backtrace-win note you don't want the 'live' backtrace, but the faulted backtrace from the crash (immediately before the runaway memory problem) What your oodles of information do -not- convey is which thread crashed (there will be a FAULT >>> comment next to the offending bytecode) and the backtrace of which function called which function called which function which then faulted. *that's* what we want to see to help you further.
I have generated a backtrace following the instructions posted at http://httpd.apache.org/dev/debugging.html#backtrace-win and have the drwtsn32.log file, as well as, the user.dmp file. I'm interested in this topic because our organization's Apache 2.2.4 installations on Windows 2003 server are setup similar to what is described above and having similar problems, what exactly are we seeking in the backtrace? It is nearly 2 MB in size. (In reply to comment #7) > The -backtrace- I refer to is spelled out at > http://httpd.apache.org/dev/debugging.html#backtrace-win > note you don't want the 'live' backtrace, but the faulted backtrace from the > crash (immediately before the runaway memory problem) > What your oodles of information do -not- convey is which thread crashed > (there will be a FAULT >>> comment next to the offending bytecode) and the > backtrace of which function called which function called which function which > then faulted. *that's* what we want to see to help you further.
I did find one "FAULT ->" entry in the drwtsn32.log file at the very bottom of all the data. I will include this 'state dump for thread' section below. The memory problem we are experiencing does not cause a crash until eventually all the available RAM is used. The process memory simply continues growing as dynamic requests come in from the time the service starts until it can grow no larger. Therefore getting a faulted backtrace from immediately before the runaway memory problem occurs may not be possible as far as I can tell. Here is the state dump section but I'm not sure if it's what is needed: *----> State Dump for Thread Id 0x6f0 <----* eax=7ffd7000 ebx=00000001 ecx=00000002 edx=00000003 esi=00000004 edi=00000005 eip=7c81a3e1 esp=08a0ffcc ebp=08a0fff4 iopl=0 nv up ei pl zr na po nc cs=001b ss=0023 ds=0023 es=0023 fs=0038 gs=0000 efl=00000246 function: ntdll!DbgBreakPoint 7c81a3ca f00fc11e lock xadd [esi],ebx 7c81a3ce 43 inc ebx 7c81a3cf 83fbff cmp ebx,0xffffffff 7c81a3d2 0f85db2e0200 jne ntdll!RtlLockHeap+0x28e (7c83d2b3) 7c81a3d8 5e pop esi 7c81a3d9 5b pop ebx 7c81a3da 33c0 xor eax,eax 7c81a3dc 5f pop edi 7c81a3dd 5d pop ebp 7c81a3de c20400 ret 0x4 FAULT ->ntdll!DbgBreakPoint: 7c81a3e1 cc int 3 7c81a3e2 c3 ret 7c81a3e3 90 nop 7c81a3e4 8bff mov edi,edi ntdll!DbgUserBreakPoint: 7c81a3e6 cc int 3 7c81a3e7 c3 ret 7c81a3e8 8bff mov edi,edi 7c81a3ea 8b442404 mov eax,[esp+0x4] 7c81a3ee cc int 3 7c81a3ef c20400 ret 0x4 *----> Stack Back Trace <----* ChildEBP RetAddr Args to Child 08a0fff4 00000000 00000000 00000000 00000000 ntdll!DbgBreakPoint *----> Raw Stack Dump <----* 0000000008a0ffcc d8 fd 83 7c 05 00 00 00 - 04 00 00 00 01 00 00 00 ...|............ 0000000008a0ffdc d0 ff a0 08 f8 8c 17 b6 - ff ff ff ff 70 82 82 7c ............p..| 0000000008a0ffec f0 fd 83 7c 00 00 00 00 - 00 00 00 00 00 00 00 00 ...|............ 0000000008a0fffc 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1000c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1001c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1002c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1003c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1004c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1005c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1006c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1007c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1008c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a1009c 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100ac 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100bc 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100cc 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100dc 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100ec 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................ 0000000008a100fc 00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00 ................
It's still not the backtrace of that specific thread (and yes, that's exactly the thread we were interested in!) Unfortunately *----> Stack Back Trace <----* ChildEBP RetAddr Args to Child 08a0fff4 00000000 00000000 00000000 00000000 ntdll!DbgBreakPoint still doesn't give us enough to go on, because you should have a series of nested functions where this fault occured, and this has no backtrace. FYI - I did not mean for you to interrupt the server while running well, but to let it crash once you have dr watson configured. It should then make you a dr watson log at the point of failure.
Insufficient information provided to diagnose; closing.