Bug 43966 - internal dummy connections blocking httpd
internal dummy connections blocking httpd
Status: RESOLVED INVALID
Product: Apache httpd-2
Classification: Unclassified
Component: All
2.2.6
PC Linux
: P1 critical (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2007-11-26 12:08 UTC by MaX Flebus
Modified: 2008-03-17 10:15 UTC (History)
1 user (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description MaX Flebus 2007-11-26 12:08:50 UTC
We have apache freezing with internal dummy connections (on https default 
vhost) apparently closing down forever.

Using mod_status I get 

__C_CKCC__CWCCWC__C__KCC_CC_CC____C__K__W____C__CCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
[...]
CCCCCCCCCCCCCCCCCCCCCCCCCCCW_CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC

That means that most of the processes are "C" Closing connection.

[...]
12-1 17525 1/449/2424 C  6.07 1275 0 0.0 8.39 40.93  127.0.0.1 web.xxx.com 
GET / HTTP/1.0 
13-1 17526 1/237/2215 C  4.65 3507 0 0.0 2.87 29.31  127.0.0.1 web.xxx.com  
GET / HTTP/1.0 

If I do strace a process, for example 17526 I see that is reading from a socket
 # strace -p 17526 
Process 17526 attached - interrupt to quit
read(356,  <unfinished ...>^C
Process 19023 detached

and if I use lsof -p 17526 to see what is fd 356 I get:
apache2 17526 apache  356r  FIFO      0,6           82466795 pipe

When every process is in status "C" the server no longer responds, and doing 
graceful or restart gets even worse for load rockets high and machine gets 
unstable.

I've found a workaround: writing rubbish in the pipe (eg: cat /etc/passwd 
> /proc/17526/fd/356 frees all the children and resumes the apache), but 
that's not a solution :)


HTTPD:
Server version: Apache/2.2.6 (Unix)
Server built:   Nov 20 2007 22:05:17
Server's Module Magic Number: 20051115:5
Server loaded:  APR 1.2.11, APR-Util 1.2.10
Compiled using: APR 1.2.11, APR-Util 1.2.10
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

running this way:
/usr/sbin/apache2 -D DEFAULT_VHOST -D PHP5 -D SSL -D INFO -d /usr/lib/apache2 -
f /etc/apache2/httpd.conf -k start

Thanks in advance for any help.
Comment 1 Attila Soki 2008-03-12 03:36:54 UTC
Same issue on Mac OS X 10.5.2

HW: XServe Xeon
Arch: x86_64
httpd 2.2.8

The version 2.2.3 runs without any problem.

I think, the bug is somewhere in the function ap_mpm_safe_kill (or apr_proc_wait), because the working version, 2.2.3 (maybe < 2.2.5) has the old kill in prefork.c and not safe_kill. but i am not sure...


Thanks for any help
Comment 2 Ruediger Pluem 2008-03-12 07:57:08 UTC
Please provide a stack backtrace of a hanging process.
Comment 3 MaX Flebus 2008-03-12 10:25:49 UTC
Can you please explain how do you want us to provide a backtrace?

MaX
Comment 4 Ruediger Pluem 2008-03-12 13:58:49 UTC
(In reply to comment #3)
> Can you please explain how do you want us to provide a backtrace?
> 
> MaX
> 

http://httpd.apache.org/dev/debugging.html
Comment 5 Attila Soki 2008-03-12 17:26:51 UTC
here is the requested backtrace:

created on mac os x 10.5.2 macbook pro intel core duo with

looks like some php related problem. without the php module works fine...


sh-3.2# gdb /usr/local/bin/httpd 58895
GNU gdb 6.3.50-20050815 (Apple version gdb-768) (Tue Oct  2 04:07:49 UTC 2007)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin"...Reading symbols for shared libraries .......... done

/usr/local/58895: No such file or directory.
Attaching to program: `/usr/local/bin/httpd', process 58895.
Reading symbols for shared libraries +++++++++.................... done

0x00007fff82976384 in read ()
(gdb) whe
#0  0x00007fff82976384 in read ()
#1  0x00000001006e90d6 in java_shutdown_library ()
#2  0x00000001006de925 in zm_shutdown_java ()
#3  0x000000010128abfb in module_destructor ()
#4  0x0000000101292621 in zend_hash_apply_deleter ()
#5  0x0000000101292778 in zend_hash_graceful_reverse_destroy ()
#6  0x0000000101287397 in zend_shutdown ()
#7  0x0000000101241fdf in php_module_shutdown ()
#8  0x0000000101242089 in php_module_shutdown_wrapper ()
#9  0x000000010131e7f1 in php_apache_child_shutdown ()
#10 0x000000010022d299 in apr_pool_destroy ()
#11 0x0000000100057eb6 in clean_child_exit ()
#12 0x00000001000584d2 in child_main ()
#13 0x000000010005869a in make_child ()
#14 0x000000010005874b in startup_children ()
#15 0x0000000100059900 in ap_mpm_run ()
#16 0x0000000100007b68 in main ()


Server version: Apache/2.2.8 (Unix)
Server built:   Mar 11 2008 16:13:40
Server's Module Magic Number: 20051115:11
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_FLOCK_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr/local"
 -D SUEXEC_BIN="/usr/local/bin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"


Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 log_config_module (static)
 env_module (static)
 unique_id_module (static)
 setenvif_module (static)
 ssl_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 status_module (static)
 autoindex_module (static)
 asis_module (static)
 cgi_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 userdir_module (static)
 alias_module (static)
 so_module (static)
 info_module (shared)
 php5_module (shared)


php 5.2.5 configuration:
'./configure' '--prefix=/usr/local'
'--with-apxs2=/usr/local/bin/apxs'
'--with-pgsql=/usr/local'
'--enable-ftp'
'--with-zlib'
'--enable-mbstring'
'--enable-sockets'
'--with-iconv'

php modules: javabridge 4.0.1 and apc



thanks.
Comment 6 Ruediger Pluem 2008-03-13 00:14:30 UTC
(In reply to comment #5)
> here is the requested backtrace:
> 
> created on mac os x 10.5.2 macbook pro intel core duo with
> 
> looks like some php related problem. without the php module works fine...
> 
> 
> sh-3.2# gdb /usr/local/bin/httpd 58895
> GNU gdb 6.3.50-20050815 (Apple version gdb-768) (Tue Oct  2 04:07:49 UTC 2007)
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "i386-apple-darwin"...Reading symbols for shared
> libraries .......... done
> 
> /usr/local/58895: No such file or directory.
> Attaching to program: `/usr/local/bin/httpd', process 58895.
> Reading symbols for shared libraries +++++++++.................... done
> 
> 0x00007fff82976384 in read ()
> (gdb) whe
> #0  0x00007fff82976384 in read ()
> #1  0x00000001006e90d6 in java_shutdown_library ()
> #2  0x00000001006de925 in zm_shutdown_java ()
> #3  0x000000010128abfb in module_destructor ()
> #4  0x0000000101292621 in zend_hash_apply_deleter ()
> #5  0x0000000101292778 in zend_hash_graceful_reverse_destroy ()
> #6  0x0000000101287397 in zend_shutdown ()
> #7  0x0000000101241fdf in php_module_shutdown ()
> #8  0x0000000101242089 in php_module_shutdown_wrapper ()
> #9  0x000000010131e7f1 in php_apache_child_shutdown ()

Your process is hanging inside mod_php. This is not an httpd bug. Please open a bug with php at http://bugs.php.net/

Comment 7 Takashi Sato 2008-03-13 00:46:55 UTC
PHP is not run by internal dummy connection in 2.2.8.
I recommend all users who faced problems about internal dummy connections update the server to 2.2.8.
Comment 8 MaX Flebus 2008-03-13 01:09:09 UTC
My idea is that the PHP module gets stuck (and so it shows up in the backtrace) while using the dummy connection AFTER something else happens.

When I write some rubbish to the pipe all the processes exit from the freeze and resume normally.
So I do not agree it's a PHP problem. Maybe java? But how to find?

Anyway I'm a little surprised this problem was simply dismissed as a non-problem after being unattended for 4 months, so I'm reopening it in the hope someone will make a better attempt at analyzing it.
Comment 9 Takashi Sato 2008-03-13 01:43:56 UTC
(In reply to comment #7)
> PHP is not run by internal dummy connection in 2.2.8.
> I recommend all users who faced problems about internal dummy connections
> update the server to 2.2.8.

My comment is very pointless. Please ignore.


(In reply to comment #8)
> So I do not agree it's a PHP problem. Maybe java? But how to find?
Your backtrace may be different from Attila's, so your backtrace may help.
Comment 10 MaX Flebus 2008-03-13 08:47:17 UTC
# gdb /usr/sbin/apache2 2295
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu"...
(no debugging symbols found)
Using host libthread_db library "/lib/libthread_db.so.1".
Attaching to program: /usr/sbin/apache2, process 2295
Reading symbols from /usr/lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcre.so.0
Reading symbols from /usr/lib64/libaprutil-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib64/libnsl.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/lib64/libgdbm.so.3...(no debugging symbols found)...done.
[...]
Reading symbols from /usr/lib64/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib64/libssl.so.0.9.7...done.
Loaded symbols for /usr/lib/libssl.so.0.9.7
Reading symbols from /usr/lib64/libcrypto.so.0.9.7...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.7
Reading symbols from /usr/lib64/php5/lib/php/extensions/no-debug-non-zts-20060613/java.so...done.
Loaded symbols for /usr/lib64/php5/lib/php/extensions/no-debug-non-zts-20060613/java.so
0x00002b8f0708f620 in read () from /lib/libc.so.6

(gdb) whe
#0  0x00002b8f0708f620 in read () from /lib/libc.so.6
#1  0x00002b8f0d34a110 in java_shutdown_library ()
   from /usr/lib64/php5/lib/php/extensions/no-debug-non-zts-20060613/java.so
#2  0x00002b8f0d347e4f in zm_shutdown_java ()
   from /usr/lib64/php5/lib/php/extensions/no-debug-non-zts-20060613/java.so
#3  0x00002b8f0a414fa1 in module_destructor () from /usr/lib/apache2/modules/libphp5.so
#4  0x00002b8f0a41b862 in ?? () from /usr/lib/apache2/modules/libphp5.so
#5  0x00002b8f0a41bad8 in zend_hash_graceful_reverse_destroy () from /usr/lib/apache2/modules/libphp5.so
#6  0x00002b8f0a411367 in zend_shutdown () from /usr/lib/apache2/modules/libphp5.so
#7  0x00002b8f0a3cf30a in php_module_shutdown () from /usr/lib/apache2/modules/libphp5.so
#8  0x00002b8f0a3cf3a9 in php_module_shutdown_wrapper () from /usr/lib/apache2/modules/libphp5.so
#9  0x00002b8f0a48f081 in ?? () from /usr/lib/apache2/modules/libphp5.so
#10 0x00002b8f0696cb5d in ?? () from /usr/lib/libapr-1.so.0
#11 0x00002b8f0696d16d in apr_pool_destroy () from /usr/lib/libapr-1.so.0
#12 0x000000000044380e in ?? ()
#13 0x0000000000443c82 in ?? ()
#14 0x0000000000443e24 in ?? ()
#15 0x0000000000443ec6 in ?? ()
#16 0x0000000000444b1d in ap_mpm_run ()
#17 0x0000000000420be7 in main ()

/usr/sbin/apache2 -V
Server version: Apache/2.2.6 (Unix)
Server built:   Nov 20 2007 23:36:41
Server's Module Magic Number: 20051115:5
Server loaded:  APR 1.2.8, APR-Util 1.2.8
Compiled using: APR 1.2.8, APR-Util 1.2.8
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/usr"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/var/run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="/var/run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
 -D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"
Comment 11 Ruediger Pluem 2008-03-13 09:51:56 UTC
Thanks for the backtrace. It shows exactly the same issue as the one by Attila Soki. It is a php issue and NOT an httpd issue. Please open a bug with php at http://bugs.php.net/.
Comment 12 Attila Soki 2008-03-13 12:33:16 UTC
hi Ruediger,

i think this is a php-java-bridge bug.
see my bugreport and patch on the javabridge mailing list:

http://sourceforge.net/mailarchive/message.php?msg_name=1205435522.2906.47.camel%40kukac

with this patch works apache fine. i hope my patch is not bugous...

but one last question: why works the same javabridge with the apache 2.2.3?


thanks
Comment 13 Ruediger Pluem 2008-03-13 12:58:06 UTC
(In reply to comment #12)
> hi Ruediger,
> 
> i think this is a php-java-bridge bug.
> see my bugreport and patch on the javabridge mailing list:
> 
> http://sourceforge.net/mailarchive/message.php?msg_name=1205435522.2906.47.camel%40kukac
> 
> with this patch works apache fine. i hope my patch is not bugous...

I don't know because I don't know the code of php-java-bridge.

> 
> but one last question: why works the same javabridge with the apache 2.2.3?

No idea.

Comment 14 Attila Soki 2008-03-15 09:57:01 UTC
hi,

i contacted the php-java-bridge developers and i get the following feedback:

linkt to the thread:
http://sourceforge.net/mailarchive/forum.php?thread_name=1205435208.2906.45.camel%40kukac&forum_name=php-java-bridge-users

On Sat, 2008-03-15 at 13:54 +0100, php-java-bridge-users@lists.sourceforge.net wrote:
Hi,
> 
> I was able to reproduce this problem with Apache .2.8 (.2.6 doesn't have this problem).
> 
> Since Apache 2.2.8 the PHP MINIT() is called exactly once and the MSHUTDOWN() function is called
> more than once, violating the PHP module contract.
> 
> In other words: PHP does not work anymore with Apache 2.2.8. Please use some other HTTP server or
> some other SAPI, for example FastCGI.
> 
> I have the gut feeling that this is a long-time PHP bug; they've probably implemented the wrong
> callback. PHP's MSHUTDOWN() shouldn't be called for each child, it should be called when the whole
> HTTP server terminates. 
> 
> 
> A quick workaround is to store the PID in the MINIT() function (see java.c) and ignore any
> MSHUTDOWN() where getpid() != pid. Something like this:
> 
> In java.c:
> int pid;
> MINIT()
> {
>   pid = getpid();
>   ...
> }
> 
> MSHUTDOWN()
> {
>   if (getpid() != pid) return SUCCESS; // work around for a Apache 2.2.8 or PHP bug.
>  ...
> }
> 
> 
> Note that all PHP modules have the same problem, so I suggest to file a bug report to either the
> PHP or the Apache team.
> 
> 
> Regards,
> Jost Boekemeier
> 
Comment 15 Attila Soki 2008-03-17 01:49:36 UTC
forgot to reopen the bug.
Comment 16 Ruediger Pluem 2008-03-17 02:26:57 UTC
(In reply to comment #15)
> forgot to reopen the bug.
> 

Why did you reopen this bug? As stated numerous times this is NO httpd bug, or do you have any new information to provide?
Comment 17 Attila Soki 2008-03-17 02:45:33 UTC
just because of the feedback from the javavridge developer team...
they says that this is an apache bug.

see my comment (#14)
Comment 18 Takashi Sato 2008-03-17 10:15:09 UTC
> I have the gut feeling that this is a long-time PHP bug; they've probably implemented the wrong
> callback. PHP's MSHUTDOWN() shouldn't be called for each child, it should be called when the whole
> HTTP server terminates. 

He thinks it is a PHP bug.