Dear all, we have a webmail platform based on Reverse Proxies + Apache + Squirrelmail. Lately we updated Apache's from 1.3.x to 2.0.54 (using standard Sarge Debian packages), since this update, we are watching that Apache reaches MaxClients often (once or twice daily), and the only one way to restore the service is to restart the daemon. Luckly, we have been able to test a "hanged-up" server, and we made several test/meassures: 1. We noticed that MaxClients works, because it's configured to manage X clients and there're X processes runing into de OS. 2. We watched that we had X established connections too. At this point everything looks fine, but there are a few strange things... 3. The connection among reverse proxy and apaches didn't carry data, just ACKs. Why isn't TimeOut rejecting this connections? 4. In order to verify who's working bad Apache or Proxies (Although the haven't been cahnged during the update), we inserted a rule into the firewall rejecting any input packet to the webserver. Result: connections became closed, but only after 45 minutes (and our TimeOut is configured in 120 seconds...!) 5. Once connections closed, we hoped that processes should be removed, and the server will start to accept new incoming requests. Not as expected, the server is still "hanged-up". Data: MDM: Prefork ii apache-common 1.3.33-6sarge3 support files for all Apache webservers ii apache2 2.0.54-5sarge1 next generation, scalable, extendable web se ii apache2-common 2.0.54-5sarge1 next generation, scalable, extendable web se ii apache2-mpm-pr 2.0.54-5sarge1 traditional model for Apache2 ii apache2-utils 2.0.54-5sarge1 utility programs for webservers ii libapache2-mod 4.3.10-19 server-side, HTML-embedded scripting languag LockFile /var/lock/apache2/accept.lock PidFile /var/run/apache2.pid Timeout 120 KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15 ServerTokens Prod <IfModule prefork.c> StartServers 10 MinSpareServers 10 MaxSpareServers 15 MaxClients 25 MaxRequestsPerChild 1000 </IfModule>
This might be a problem with PHP / Squirrelmail. Please try to obtain backtraces from "hanging" processes (see http://httpd.apache.org/dev/debugging.html). You might misread the meaning of Timeout in httpd. Timeout here means the maximum amount of time httpd waits to *write* data to or to *read* data from the network. If you are using PHP via mod_php (running as a CGI is a different story) and your php script simply generates no output httpd will keep the connection open forever which works as designed as there is nothing to read from the network for httpd (we are sending the response) and there is nothing to write to the network (since the script produces no output). The max_execution_time setting in PHP is used to prevent such long running scripts. I cannot tell you if this works reliable.
Additional information.... This is the variable of the max_execution_time... php.ini:max_execution_time = 30 ; Maximum execution time of each script, in seconds As you could see, after 30 seconds the execution should be turned off, and processes being released, but it didn't happen.
Debug info... #ps -ef | grep apache root 5224 1 0 May20 ? 00:00:02 /usr/sbin/apache2 -k start -DSSL Debian- 5171 5224 0 May21 ? 00:00:00 /usr/sbin/apache2 -k start -DSSL Debian- 5183 5224 0 May21 ? 00:00:01 /usr/sbin/apache2 -k start -DSSL // *************************************************** // PSTACK // *************************************************** It's seems that there aren't symbols into binaries included into Debian package, so, there's no inof here, isn't there? #pstack 5224 5224: /usr/sbin/apache2 -k start -DSSL (No symbols found) 0x4042b9f8: ???? (f4240, 0, bffff118, 1, 80b00a8, c) 0x08084598: ???? (bffff118, bffff11c, bffff120, 80b00a8, 80b4b78, 809c495) + 40 0x08077042: ???? (80b00a8, 80e6180, 80b4b78, 80b4b78, bffff194, 403651a8) + 40 0x0807da8d: ???? (4, bffff234, bffff248, 0, 4048bf40, 40016540) + 40 0x4036f974: ???? (807d3c0, 4, bffff234, 80948e0, 8094940, 4000c380) + 40000dd8 # pstack 5183 5183: /usr/sbin/apache2 -k start -DSSL (No symbols found) 0x40424ca1: ???? (8225c8c, 824dae4, bfffa048, 40726945, 8235574, 40883dc1) + 80 0x406713eb: ???? (408cc3f0, 824dae4, bfffa0b4, bfffa0b8, 824dae4, 82a95ec) + 10 0x4066d656: ???? (82a95ec, 20, a, bfffa100, 0, bfffa104) + 30 0x4066ebe0: ???? (408c8274, 8c) 0x40670660: ???? (0, 824db1c, 0, 0, 4, bfffa14c) + 2a0 0x407331c0: ???? (824d9e8, 0, 0, 4071a600, 0, 0) + bf0 0x40732f25: ???? (824da4c, 8, 2d, bfffb808, 4, 0) + 8e0 0x40734614: ???? (821634c, 8, 2b, bfffca88, 4, 0) + 1260 0x40734614: ???? (8227c94, 2, 408d03e0, 827e990, 1, bfffcbc4) + 10 0x40722db1: ???? (8, 0, 3, 0, bfffedc0, 0) + 2120 0x406f4ebf: ???? (bfffedc0, 820ed10, 40737c30, 80606c8, 0, bfffebf8) + 1b0 0x4073834f: ???? (820c5b0, 80b5868, 0, 820c5b0, 820c5b0, 8128cb0) 0x08078375: ???? (820c5b0, 3b, 81e792c, 820c5b0, 0, 81e78e8) + 20 0x08078980: ???? (820c5b0, 0, 4, 820c5b0, 81e78e8, 820c5b0) 0x08069c6a: ???? (820c5b0, 4, 820c5b0, 808370c, 81e78e8, 81e7810) + 10 0x0806512d: ???? (81e78e8, 81e7810, 2, 81e5890, 81e78e8, 8206560) 0x080835c5: ???? (81e78e8, 81e7810, 81e7810, 18, 81e5848, 8206560) + 30 0x08076974: ???? (18, 1, 1, 0, 1, 816f120) + 10 0x08076ac7: ???? (80b4b78, 18, 40399c4, 40016ed8, 805957d, bffff074) + c0 0x08076d59: ???? (80b00a8, bffff11c, bffff120, 80b00a8, 80b4b78, 809c495) + 40 0x08077445: ???? (80b00a8, 80e6180, 80b4b78, 80b4b78, bffff194, 403651a8) + 40 0x0807da8d: ???? (4, bffff234, bffff248, 0, 4048bf40, 40016540) + 40 0x4036f974: ???? (807d3c0, 4, bffff234, 80948e0, 8094940, 4000c380) + 40000dd8 // *************************************************** // STRACE // *************************************************** The process outputs these calls, and it doesn't change when I try making a connection to the httpd process # strace -p 5224 Process 5224 attached - interrupt to quit select(0, NULL, NULL, NULL, {0, 920000}) = 0 (Timeout) waitpid(-1, 0xbffff090, WNOHANG|WUNTRACED) = 0 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) waitpid(-1, 0xbffff090, WNOHANG|WUNTRACED) = 0 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) waitpid(-1, 0xbffff090, WNOHANG|WUNTRACED) = 0 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) waitpid(-1, 0xbffff090, WNOHANG|WUNTRACED) = 0 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) waitpid(-1, 0xbffff090, WNOHANG|WUNTRACED) = 0 select(0, NULL, NULL, NULL, {1, 0} <unfinished ...> Process 5224 detached Child seems to be more interesting. It looks like locked... # strace -p 5183 Process 5183 attached - interrupt to quit flock(27, LOCK_EX Must I perform any other debug command? Thanks in advance.
Without symbols it is nearly impossible to track this. Furthermore please use gdb to obtain a backtrace as described on http://httpd.apache.org/dev/debugging.html. You only need to analyse the stuck non-root processes. The process running under root is of no interest here.
While I try to execute gdb, another detail... This is the strace of a child process on a server which's at this moment working fine. Here there's not lock. Is this a clue? # strace -p 22084 Process 22084 attached - interrupt to quit semop(622594, 0x4030e7ec, 1 <unfinished ...> Process 22084 detached
Questions about gdb: 1. Should I recompile with symbols before using gdb? 2. Because I can't follow some steps to make a fail, I suppose I should wait until all my child process get hanged (as at this moment) and then execute gdb against a child-pid, don't I? 3. Once in debugging mode, which variables do we need to get? # gdb /usr/sbin/apache2 5183 GNU gdb 6.3-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...(no debugging symbols found) Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /usr/sbin/apache2, process 5183 (no debugging symbols found) `system-supplied DSO at 0xffffe000' has disappeared; keeping its symbols. Reading symbols from /lib/tls/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libdl.so.2 Reading symbols from /lib/tls/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libcrypt.so.1 Reading symbols from /usr/lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpcre.so.3 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.7... (no debugging symbols found)...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.7 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.7 Reading symbols from /usr/lib/libaprutil-0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libaprutil-0.so.0 Reading symbols from /usr/lib/libldap_r.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libldap_r.so.2 Reading symbols from /usr/lib/liblber.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/lib/liblber.so.2 Reading symbols from /usr/lib/libdb-4.2.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb-4.2.so Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /usr/lib/libapr-0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libapr-0.so.0 Reading symbols from /lib/tls/librt.so.1... (no debugging symbols found)...done. Loaded symbols for /lib/tls/librt.so.1 Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/tls/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libnsl.so.1 Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] [New Thread 1079503776 (LWP 5183)] Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/tls/libc.so.6... (no debugging symbols found)...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libresolv.so.2 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgnutls.so.11... (no debugging symbols found)...done. Loaded symbols for /usr/lib/libgnutls.so.11 Reading symbols from /usr/lib/libtasn1.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libtasn1.so.2 Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgcrypt.so.11 Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgpg-error.so.0 Reading symbols from /lib/tls/libnss_files.so.2... (no debugging symbols found)...done. Loaded symbols for /lib/tls/libnss_files.so.2 Reading symbols from /usr/lib/apache2/modules/mod_mime_magic.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_mime_magic.so Reading symbols from /usr/lib/apache2/modules/mod_expires.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_expires.so Reading symbols from /usr/lib/apache2/modules/mod_deflate.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_deflate.so Reading symbols from /usr/lib/apache2/modules/mod_vhost_alias.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_vhost_alias.so Reading symbols from /usr/lib/apache2/modules/mod_userdir.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_userdir.so Reading symbols from /usr/lib/apache2/modules/mod_rewrite.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_rewrite.so Reading symbols from /usr/lib/apache2/modules/mod_ssl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_ssl.so Reading symbols from /usr/lib/apache2/modules/libphp4.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/libphp4.so Reading symbols from /usr/lib/libzzip-0.so.12...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libzzip-0.so.12 Reading symbols from /usr/lib/libbz2.so.1.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libbz2.so.1.0 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3... (no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/apache2/modules/mod_unique_id.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_unique_id.so Reading symbols from /usr/lib/php4/20020429/ldap.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/php4/20020429/ldap.so Reading symbols from /usr/lib/php4/20020429/apc.so...done. Loaded symbols for /usr/lib/php4/20020429/apc.so Reading symbols from /lib/tls/libnss_compat.so.2...done. Loaded symbols for /lib/tls/libnss_compat.so.2 Reading symbols from /lib/tls/libnss_nis.so.2...done. Loaded symbols for /lib/tls/libnss_nis.so.2 Reading symbols from /lib/libnss_ldap.so.2...done. Loaded symbols for /lib/libnss_ldap.so.2 Reading symbols from /usr/lib/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib/sasl2/libsasldb.so.2 Reading symbols from /lib/tls/libnss_dns.so.2...done. Loaded symbols for /lib/tls/libnss_dns.so.2 0x40424ca1 in flock () from /lib/tls/libc.so.6 (gdb)
(In reply to comment #6) > Questions about gdb: > > 1. Should I recompile with symbols before using gdb? That would be helpful, but is not strictly needed. You can make a first try with your current binary. > 2. Because I can't follow some steps to make a fail, I suppose I should wait > until all my child process get hanged (as at this moment) and then execute gdb > against a child-pid, don't I? Yes, attachg gdb to a hanging process (aka live backtrace). See http://httpd.apache.org/dev/debugging.html#backtrace > 3. Once in debugging mode, which variables do we need to get? Just do thread apply all bt full and we get everything that is there and that is needed for the start. If more is needed it needs to be specified later.
Thanks a lot for your clarifications. Here it is... # gdb /usr/sbin/apache2 5183 GNU gdb 6.3-debian Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-linux"...(no debugging symbols found) Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /usr/sbin/apache2, process 5183 (no debugging symbols found) `system-supplied DSO at 0xffffe000' has disappeared; keeping its symbols. Reading symbols from /lib/tls/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libdl.so.2 Reading symbols from /lib/tls/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libcrypt.so.1 Reading symbols from /usr/lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpcre.so.3 Reading symbols from /usr/lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libz.so.1 Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.7... (no debugging symbols found)...done. Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.7 Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.7 Reading symbols from /usr/lib/libaprutil-0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libaprutil-0.so.0 Reading symbols from /usr/lib/libldap_r.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libldap_r.so.2 Reading symbols from /usr/lib/liblber.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/lib/liblber.so.2 Reading symbols from /usr/lib/libdb-4.2.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdb-4.2.so Reading symbols from /usr/lib/libexpat.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libexpat.so.1 Reading symbols from /usr/lib/libapr-0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libapr-0.so.0 Reading symbols from /lib/tls/librt.so.1... (no debugging symbols found)...done. Loaded symbols for /lib/tls/librt.so.1 Reading symbols from /lib/tls/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libm.so.6 Reading symbols from /lib/tls/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libnsl.so.1 Reading symbols from /lib/tls/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] [New Thread 1079503776 (LWP 5183)] Loaded symbols for /lib/tls/libpthread.so.0 Reading symbols from /lib/tls/libc.so.6... (no debugging symbols found)...done. Loaded symbols for /lib/tls/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/tls/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/libresolv.so.2 Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /usr/lib/libgnutls.so.11... (no debugging symbols found)...done. Loaded symbols for /usr/lib/libgnutls.so.11 Reading symbols from /usr/lib/libtasn1.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libtasn1.so.2 Reading symbols from /usr/lib/libgcrypt.so.11...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgcrypt.so.11 Reading symbols from /usr/lib/libgpg-error.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgpg-error.so.0 Reading symbols from /lib/tls/libnss_files.so.2... (no debugging symbols found)...done. Loaded symbols for /lib/tls/libnss_files.so.2 Reading symbols from /usr/lib/apache2/modules/mod_mime_magic.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_mime_magic.so Reading symbols from /usr/lib/apache2/modules/mod_expires.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_expires.so Reading symbols from /usr/lib/apache2/modules/mod_deflate.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_deflate.so Reading symbols from /usr/lib/apache2/modules/mod_vhost_alias.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_vhost_alias.so Reading symbols from /usr/lib/apache2/modules/mod_userdir.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_userdir.so Reading symbols from /usr/lib/apache2/modules/mod_rewrite.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_rewrite.so Reading symbols from /usr/lib/apache2/modules/mod_ssl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_ssl.so Reading symbols from /usr/lib/apache2/modules/libphp4.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/libphp4.so Reading symbols from /usr/lib/libzzip-0.so.12...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libzzip-0.so.12 Reading symbols from /usr/lib/libbz2.so.1.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libbz2.so.1.0 Reading symbols from /usr/lib/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgssapi_krb5.so.2 Reading symbols from /usr/lib/libkrb5.so.3... (no debugging symbols found)...done. Loaded symbols for /usr/lib/libkrb5.so.3 Reading symbols from /usr/lib/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libk5crypto.so.3 Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcom_err.so.2 Reading symbols from /usr/lib/apache2/modules/mod_unique_id.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apache2/modules/mod_unique_id.so Reading symbols from /usr/lib/php4/20020429/ldap.so... (no debugging symbols found)...done. Loaded symbols for /usr/lib/php4/20020429/ldap.so Reading symbols from /usr/lib/php4/20020429/apc.so...done. Loaded symbols for /usr/lib/php4/20020429/apc.so Reading symbols from /lib/tls/libnss_compat.so.2...done. Loaded symbols for /lib/tls/libnss_compat.so.2 Reading symbols from /lib/tls/libnss_nis.so.2...done. Loaded symbols for /lib/tls/libnss_nis.so.2 Reading symbols from /lib/libnss_ldap.so.2...done. Loaded symbols for /lib/libnss_ldap.so.2 Reading symbols from /usr/lib/sasl2/libsasldb.so.2...done. Loaded symbols for /usr/lib/sasl2/libsasldb.so.2 Reading symbols from /lib/tls/libnss_dns.so.2...done. Loaded symbols for /lib/tls/libnss_dns.so.2 0x40424ca1 in flock () from /lib/tls/libc.so.6 (gdb) where #0 0x40424ca1 in flock () from /lib/tls/libc.so.6 #1 0x40670f87 in zm_info_session () from /usr/lib/apache2/modules/libphp4.so #2 0x406713eb in ps_read_files () from /usr/lib/apache2/modules/libphp4.so #3 0x4066d656 in php_session_create_id () from /usr/lib/apache2/modules/libphp4.so #4 0x4066ebe0 in php_session_start () from /usr/lib/apache2/modules/libphp4.so #5 0x40670660 in zif_session_start () from /usr/lib/apache2/modules/libphp4.so #6 0x407331c0 in execute () from /usr/lib/apache2/modules/libphp4.so #7 0x40732f25 in execute () from /usr/lib/apache2/modules/libphp4.so #8 0x40734614 in execute () from /usr/lib/apache2/modules/libphp4.so #9 0x40734614 in execute () from /usr/lib/apache2/modules/libphp4.so #10 0x40722db1 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp4.so #11 0x406f4ebf in php_execute_script () from /usr/lib/apache2/modules/libphp4.so #12 0x4073834f in execute () from /usr/lib/apache2/modules/libphp4.so #13 0x08078375 in ap_run_handler () #14 0x08078980 in ap_invoke_handler () #15 0x08069c6a in ap_process_request () #16 0x0806512d in _start () #17 0x0820c5b0 in ?? () #18 0x00000004 in ?? () #19 0x0820c5b0 in ?? () #20 0x0808370c in ap_run_pre_connection () #21 0x080835c5 in ap_run_process_connection () #22 0x08076974 in ap_graceful_stop_signalled () #23 0x08076ac7 in ap_graceful_stop_signalled () #24 0x08076d59 in ap_graceful_stop_signalled () #25 0x08077445 in ap_mpm_run () #26 0x0807da8d in main () (gdb) thread apply all bt full Thread 1 (Thread 1079503776 (LWP 5183)): #0 0x40424ca1 in flock () from /lib/tls/libc.so.6 No symbol table info available. #1 0x40670f87 in zm_info_session () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #2 0x406713eb in ps_read_files () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #3 0x4066d656 in php_session_create_id () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #4 0x4066ebe0 in php_session_start () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #5 0x40670660 in zif_session_start () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #6 0x407331c0 in execute () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #7 0x40732f25 in execute () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #8 0x40734614 in execute () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #9 0x40734614 in execute () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #10 0x40722db1 in zend_execute_scripts () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #11 0x406f4ebf in php_execute_script () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #12 0x4073834f in execute () from /usr/lib/apache2/modules/libphp4.so No symbol table info available. #13 0x08078375 in ap_run_handler () No symbol table info available. #14 0x08078980 in ap_invoke_handler () No symbol table info available. #15 0x08069c6a in ap_process_request () No symbol table info available. #16 0x0806512d in _start () No symbol table info available. #17 0x0820c5b0 in ?? () No symbol table info available. #18 0x00000004 in ?? () No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #19 0x0820c5b0 in ?? () No symbol table info available. #20 0x0808370c in ap_run_pre_connection () No symbol table info available. #21 0x080835c5 in ap_run_process_connection () No symbol table info available. #22 0x08076974 in ap_graceful_stop_signalled () No symbol table info available. #23 0x08076ac7 in ap_graceful_stop_signalled () No symbol table info available. #24 0x08076d59 in ap_graceful_stop_signalled () No symbol table info available. #25 0x08077445 in ap_mpm_run () No symbol table info available. #26 0x0807da8d in main () No symbol table info available. #0 0x40424ca1 in flock () from /lib/tls/libc.so.6 (gdb) quit The program is running. Quit anyway (and detach it)? (y or n) y Detaching from program: /usr/sbin/apache2, process 5183
Thanks for the backtrace. First of all the backtrace proves my initial thoughts: You are stuck in PHP land. This might be a bug in PHP, in your PHP configuration or in your PHP application. I don't know. So please try to report this to bugs.php.net or to squirrelmail to obtain further help. From a first glance I would guess that the problem is related to PHP's session management via files and the needed locks during these operations. It might make sense for you to obtain backtraces from *any* child process in this situation to get a better impression 1. if all processes are waiting for one specific process 2. what the process holding the lock is actually doing. You might be able to find the process that holds the lock via lsof | egrep "[0-9][a-zA-Z][a-zA-Z] "
Thanks a lot for your effort. Regards.