Bug 41030 - Worker segmentation fault under load
Summary: Worker segmentation fault under load
Status: RESOLVED DUPLICATE of bug 44402
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mpm_worker (show other bugs)
Version: 2.2.3
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-11-23 11:59 UTC by Jesse Pool
Modified: 2010-10-06 19:06 UTC (History)
1 user (show)


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Pool 2006-11-23 11:59:47 UTC 
ERROR: /usr/local/apache2_worker/logs/error_log
[Wed Nov 22 10:45:43 2006] [notice] child pid 14801 exit signal Segmentation
fault (11), possible coredump in /tmp/apache2-dump

SENARIO:
I've been loading Apache using Webstone (an open source tool). It only makes
HTTP 1.0 requests. I've set it to repeatedly GET the same 500 byte file using
10 client threads on a remote machine (connected through a 100 Mbit switch). I
get the error log message after a few minutes (which translates to a few
hundred thousand requests). Apache is running 1 process with 10 worker
threads (although, I also tried it with the default number of
processes/threads and still observed a SIGSEGV).

SYSTEM: 
- Intel Core 2 6400 @ 2.13GHz 
- Fedora Core 5 (default DVD install)
  + Linux 2.6.18 (default compile from kernel.org)

COMPILE:
[root@localhost httpd-2.2.3_worker]$ ./configure \
> --prefix=/usr/local/apache2_worker \
> --with-mpm=worker

[root@localhost linux-2.6.16]$ /usr/local/apache2_worker/bin/httpd -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_include.c
  mod_filter.c
  mod_log_config.c
  mod_env.c
  mod_setenvif.c
  worker.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_negotiation.c
  mod_dir.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_so.c

CONFIGURATION MODIFICATIONS:
#
# This will give us somthing to debug with in the case of a crash.
#
CoreDumpDirectory /tmp/apache2-dump

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    StartServers          1
    MaxClients           10
    MinSpareThreads      10
    MaxSpareThreads      10
    ThreadsPerChild      10
    MaxRequestsPerChild   0
</IfModule>

CORE DUMP:
I don't think this is a very useful backtrace, but never-the-less, here it is. 
I haven't been able to generate one with more symbol information on my system. 
Any advice on getting a more complete trace would be great. -- This is very 
reproducible. 

[root@localhost local]$  gdb /usr/local/apache2_worker/bin/httpd
/tmp/apache2-dump/core.14801
GNU gdb Red Hat Linux (6.3.0.0-1.122rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db
library "/lib/libthread_db.so.1".

Reading symbols from shared object read from target memory...done.
Loaded system supplied DSO at 0xffffe000
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.

warning: svr4_current_sos: Can't read pathname for load map: Input/output
error

Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /usr/local/apache2_worker/lib/libaprutil-1.so.0...done.
Loaded symbols for /usr/local/apache2_worker/lib/libaprutil-1.so.0
Reading symbols from /usr/lib/libsqlite3.so.0...done.
Loaded symbols for /usr/lib/libsqlite3.so.0
Reading symbols from /lib/libexpat.so.0...done.
Loaded symbols for /lib/libexpat.so.0
Reading symbols from /usr/local/apache2_worker/lib/libapr-1.so.0...done.
Loaded symbols for /usr/local/apache2_worker/lib/libapr-1.so.0
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  0xb7f09410 in ?? ()
(gdb) where
#0  0xb7f09410 in ?? ()
#1  0xbfa38a68 in ?? ()
#2  0x00000001 in ?? ()
#3  0xbfa38a67 in ?? ()
#4  0x00cc2b8b in __read_nocancel () from /lib/libpthread.so.0
#5  0x080a17d4 in ap_mpm_pod_check (pod=0x816bd40) at pod.c:54
#6  0x0809f426 in child_main (child_num_arg=Variable "child_num_arg" is not
available.
) at worker.c:1258
#7  0x0809f5a2 in make_child (s=0x80cdbe8, slot=0) at worker.c:1341
#8  0x0809f667 in startup_children (number_to_start=1) at worker.c:1375
#9  0x080a0757 in ap_mpm_run (_pconf=0x80c70a8, plog=0x81051a0, s=0x80cdbe8)
at worker.c:1725
#10 0x08062335 in main (argc=135024928, argv=0x0) at main.c:717
(gdb) thread apply all bt full

Thread 12 (process 14803):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb7674168 in ?? ()
No symbol table info available.
#2  0x000007d0 in ?? ()
No symbol table info available.
#3  0x00000001 in ?? ()
No symbol table info available.
#4  0x00bffcac in poll () from /lib/libc.so.6
No symbol table info available.
#5  0xb7edc6b4 in apr_wait_for_io_or_timeout (f=0x0, s=0x8195808, for_read=1)
at support/unix/waitio.c:51
        pfd = {fd = 10, events = 1, revents = 17}
        rc = 1
        timeout = 2000
#6  0xb7ed7a40 in apr_socket_recv (sock=0x8195808, buf=0xb76741f4
"@{&#1073;\030Bg&#65533;z&#65533;&#65533;&#65533;", len=0xb76743f4) at network_io/unix/sendrecv.c:87
        rv = Variable "rv" is not available.

Thread 11 (process 14804):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb6e73168 in ?? ()
No symbol table info available.
#2  0x000007d0 in ?? ()
No symbol table info available.
#3  0x00000001 in ?? ()
No symbol table info available.
#4  0x00bffcac in poll () from /lib/libc.so.6
No symbol table info available.
#5  0xb7edc6b4 in apr_wait_for_io_or_timeout (f=0x0, s=0x819d9e8, for_read=1)
at support/unix/waitio.c:51
        pfd = {fd = 20, events = 1, revents = 0}
        rc = -4
        timeout = 2000
#6  0xb7ed7a40 in apr_socket_recv (sock=0x819d9e8, buf=0xb6e731f4
"&#65533;\220\033\b\0302&#65533;&#65533;z&#65533;&#65533;&#65533;", len=0xb6e733f4) at network_io/unix/sendrecv.c:87
        rv = Variable "rv" is not available.

Thread 10 (process 14805):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb6672168 in ?? ()
No symbol table info available.
#2  0x000007d0 in ?? ()
No symbol table info available.
#3  0x00000001 in ?? ()
No symbol table info available.
#4  0x00bffcac in poll () from /lib/libc.so.6
No symbol table info available.
#5  0xb7edc6b4 in apr_wait_for_io_or_timeout (f=0x0, s=0x81998f8, for_read=1)
at support/unix/waitio.c:51
        pfd = {fd = 17, events = 1, revents = 0}
        rc = -4
        timeout = 2000
#6  0xb7ed7a40 in apr_socket_recv (sock=0x81998f8, buf=0xb66721f4
"\210&#65533;\032\b\030\"g&#65533;z&#65533;&#65533;&#65533;", len=0xb66723f4) at network_io/unix/sendrecv.c:87
---Type <return> to continue, or q <return> to quit---
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb5e71168 in ?? ()
No symbol table info available.
#2  0x000007d0 in ?? ()
No symbol table info available.
#3  0x00000001 in ?? ()
No symbol table info available.
#4  0x00bffcac in poll () from /lib/libc.so.6
No symbol table info available.
#5  0xb7edc6b4 in apr_wait_for_io_or_timeout (f=0x0, s=0x81bdad8, for_read=1)
at support/unix/waitio.c:51
        pfd = {fd = 12, events = 1, revents = 17}
        rc = 1
        timeout = 2000
#6  0xb7ed7a40 in apr_socket_recv (sock=0x81bdad8, buf=0xb5e711f4
"\200{&#65533;&#65533;\030\022&#65533;&#65533;z&#65533;&#65533;&#65533;", len=0xb5e713f4) at network_io/unix/sendrecv.c:87
        rv = Variable "rv" is not available.

Thread 8 (process 14807):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb56703e8 in ?? ()
No symbol table info available.
#2  0x0043311d in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.

Thread 7 (process 14808):
#0  0x08090cbb in find_ct (r=0xb1b0a4e8) at mod_mime.c:781
        conf = (mime_dir_config *) 0x80ffa68
        exception_list = (apr_array_header_t *) 0xb1b0b9b8
        ext = Variable "ext" is not available.

Thread 6 (process 14809):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb466e3e8 in ?? ()
No symbol table info available.
#2  0x0043311c in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.

Thread 5 (process 14810):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb3e6d3e8 in ?? ()
No symbol table info available.
#2  0x0043311b in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.

Thread 4 (process 14811):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb366c098 in ?? ()
No symbol table info available.
#2  0x0000000b in ?? ()
No symbol table info available.
#3  0x0000000b in ?? ()
No symbol table info available.
#4  0x00b69546 in kill () from /lib/libc.so.6
No symbol table info available.
#5  0x0807c7dd in sig_coredump (sig=14801) at mpm_common.c:1170
No locals.
#6  0xb7f09420 in ?? ()
No symbol table info available.
#7  0x0000000b in ?? ()
No symbol table info available.
#8  0x00000033 in ?? ()
No symbol table info available.
#9  0x00000000 in ?? ()
No symbol table info available.

Thread 3 (process 14812):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb2e6b3e8 in ?? ()
No symbol table info available.
#2  0x004330c0 in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.

Thread 2 (process 14813):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xb266a3d8 in ?? ()
No symbol table info available.
#2  0x0005c97b in ?? ()
No symbol table info available.
#3  0x00000000 in ?? ()
No symbol table info available.

Thread 1 (process 14801):
#0  0xb7f09410 in ?? ()
No symbol table info available.
#1  0xbfa38a68 in ?? ()
No symbol table info available.
#2  0x00000001 in ?? ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#3  0xbfa38a67 in ?? ()
No symbol table info available.
#4  0x00cc2b8b in __read_nocancel () from /lib/libpthread.so.0
No symbol table info available.
#5  0x080a17d4 in ap_mpm_pod_check (pod=0x816bd40) at pod.c:54
        c = Variable "c" is not available.
(gdb)

ADDITIONAL:
On one occasion, while running the same scenario that is described above, I hit
a double-free. Bellow is what appeared in the error_log. This only happened
once and I have not been able to reproduce to get a core dump. -- related? 

[Thu Nov 23 13:34:23 2006] [notice] Apache/2.2.3 (Unix) configured -- resuming
normal operations
[Thu Nov 23 13:34:28 2006] [error] server reached MaxClients setting, consider
raising the MaxClients setting
[Thu Nov 23 13:36:16 2006] [notice] child pid 2102 exit signal Segmentation
fault (11)
[Thu Nov 23 13:36:36 2006] [notice] child pid 2429 exit signal Segmentation
fault (11)
[Thu Nov 23 13:36:40 2006] [notice] child pid 2443 exit signal Segmentation
fault (11)
[Thu Nov 23 13:37:30 2006] [notice] child pid 2457 exit signal Segmentation
fault (11)
[Thu Nov 23 13:38:22 2006] [notice] child pid 2473 exit signal Segmentation
fault (11)
[Thu Nov 23 13:40:03 2006] [notice] child pid 2490 exit signal Segmentation
fault (11)
[Thu Nov 23 13:42:11 2006] [error] (22)Invalid argument: apr_socket_accept:
(client socket)
*** glibc detected *** /usr/local/apache2/bin/httpd: double free or corruption
(out): 0x081704f8 ***
======= Backtrace: =========
/lib/libc.so.6[0xba4f18]
/lib/libc.so.6(__libc_free+0x79)[0xba841d]
/usr/local/apache2_worker/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)[0xb7fa5e6d]
/usr/local/apache2_worker/lib/libapr-1.so.0(apr_pool_destroy+0x195)[0xb7fa65a5]
/usr/local/apache2/bin/httpd[0x80a1699]
/usr/local/apache2_worker/lib/libapr-1.so.0[0xb7fa5ccd]
/usr/local/apache2_worker/lib/libapr-1.so.0(apr_pool_destroy+0x3d)[0xb7fa644d]
/usr/local/apache2/bin/httpd[0x809f0f4]
/usr/local/apache2/bin/httpd[0x809f3c0]
/usr/local/apache2/bin/httpd[0x809f5a2]
/usr/local/apache2/bin/httpd(ap_mpm_run+0x887)[0x80a0377]
/usr/local/apache2/bin/httpd(main+0x825)[0x8062335]
/lib/libc.so.6(__libc_start_main+0xdc)[0xb567e4]
/usr/local/apache2/bin/httpd(apr_bucket_mmap_make+0x61)[0x8061561]
======= Memory map: ========
0019e000-001fa000 r-xp 00000000 fe:00 13162567   /usr/lib/libsqlite3.so.0.8.6
001fa000-001fc000 rw-p 0005b000 fe:00 13162567   /usr/lib/libsqlite3.so.0.8.6
00289000-002a8000 r-xp 00000000 fe:00 46892455   /lib/libexpat.so.0.5.0
002a8000-002aa000 rw-p 0001e000 fe:00 46892455   /lib/libexpat.so.0.5.0
0051b000-00526000 r-xp 00000000 fe:00 46892444
/lib/libgcc_s-4.1.0-20060304.so.1
00526000-00527000 rw-p 0000a000 fe:00 46892444
/lib/libgcc_s-4.1.0-20060304.so.1
0084f000-00855000 r-xp 00000000 fe:00 46892451   /lib/librt-2.4.so
00855000-00856000 r--p 00005000 fe:00 46892451   /lib/librt-2.4.so
00856000-00857000 rw-p 00006000 fe:00 46892451   /lib/librt-2.4.so
00b41000-00c6d000 r-xp 00000000 fe:00 46892441   /lib/libc-2.4.so
00c6d000-00c70000 r--p 0012b000 fe:00 46892441   /lib/libc-2.4.so
00c70000-00c71000 rw-p 0012e000 fe:00 46892441   /lib/libc-2.4.so
00c71000-00c74000 rw-p 00c71000 00:00 0
00c76000-00c78000 r-xp 00000000 fe:00 46892442   /lib/libdl-2.4.so
00c78000-00c79000 r--p 00001000 fe:00 46892442   /lib/libdl-2.4.so
00c79000-00c7a000 rw-p 00002000 fe:00 46892442   /lib/libdl-2.4.so
00c7c000-00c9f000 r-xp 00000000 fe:00 46892443   /lib/libm-2.4.so
00c9f000-00ca0000 r--p 00022000 fe:00 46892443   /lib/libm-2.4.so
00ca0000-00ca1000 rw-p 00023000 fe:00 46892443   /lib/libm-2.4.so
00cb8000-00cc8000 r-xp 00000000 fe:00 46892445   /lib/libpthread-2.4.so
00cc8000-00cc9000 r--p 0000f000 fe:00 46892445   /lib/libpthread-2.4.so
00cc9000-00cca000 rw-p 00010000 fe:00 46892445   /lib/libpthread-2.4.so
00cca000-00ccc000 rw-p 00cca000 00:00 0
0226e000-02273000 r-xp 00000000 fe:00 46892449   /lib/libcrypt-2.4.so
02273000-02274000 r--p 00004000 fe:00 46892449   /lib/libcrypt-2.4.so
02274000-02275000 rw-p 00005000 fe:00 46892449   /lib/libcrypt-2.4.so
02275000-0229c000 rw-p 02275000 00:00 0
08048000-080bb000 r-xp 00000000 fe:00 13861836
/usr/local/apache2_worker/bin/httpd
080bb000-080be000 rw-p 00073000 fe:00 13861836
/usr/local/apache2_worker/bin/httpd
080be000-08204000 rw-p 080be000 00:00 0          [heap]
b1e00000-b1e21000 rw-p b1e00000 00:00 0
b1e21000-b1f00000 ---p b1e21000 00:00 0
b273e000-b273f000 ---p b273e000 00:00 0
b273f000-b2f3f000 rw-p b273f000 00:00 0
b2f3f000-b2f40000 ---p b2f3f000 00:00 0
b2f40000-b3740000 rw-p b2f40000 00:00 0
b3740000-b3741000 ---p b3740000 00:00 0
b3741000-b3f41000 rw-p b3741000 00:00 0
b3f41000-b3f42000 ---p b3f41000 00:00 0
b3f42000-b4742000 rw-p b3f42000 00:00 0
b7f49000-b7f82000 rw-s 00000000 00:07 1019868    /dev/zero (deleted)
b7f82000-b7f8b000 r-xp 00000000 fe:00 46891049   /lib/libnss_files-2.4.so
b7f8b000-b7f8c000 r--p 00008000 fe:00 46891049   /lib/libnss_files-2.4.so
b7f8c000-b7f8d000 rw-p 00009000 fe:00 46891049   /lib/libnss_files-2.4.so
b7f8d000-b7f8f000 rw-p b7f8d000 00:00 0
b7f8f000-b7f91000 r-xp 00000000 fe:00 46892462   /lib/libuuid.so.1.2
b7f91000-b7f92000 rw-p 00002000 fe:00 46892462   /lib/libuuid.so.1.2
b7f92000-b7fb4000 r-xp 00000000 fe:00 13861773
/usr/local/apache2_worker/lib/libapr-1.so.0.2.7
b7fb4000-b7fb5000 rw-p 00022000 fe:00 13861773
/usr/local/apache2_worker/lib/libapr-1.so.0.2.7
b7fb5000-b7fb6000 rw-p b7fb5000 00:00 0
b7fb6000-b7fcb000 r-xp 00000000 fe:00 13861815
/usr/local/apache2_worker/lib/libaprutil-1.so.0.2.7
b7fcb000-b7fcc000 rw-p 00014000 fe:00 13861815
/usr/local/apache2_worker/lib/libaprutil-1.so.0.2.7
b7fdb000-b7fdc000 rw-p b7fdb000 00:00 0
b7fdc000-b7ff5000 r-xp 00000000 fe:00 46892440   /lib/ld-2.4.so
b7ff5000-b7ff6000 r--p 00018000 fe:00 46892440   /lib/ld-2.4.so
b7ff6000-b7ff7000 rw-p 00019000 fe:00 46892440   /lib/ld-2.4.so
bf8e2000-bf8f7000 rw-p bf8e2000 00:00 0          [stack]
ffffe000-fffff000 ---p 00000000 00:00 0          [vdso]
[Thu Nov 23 13:42:11 2006] [notice] child pid 2510 exit signal Aborted (6)
Comment 1 Jesse Pool 2006-11-24 08:04:35 UTC 
I've put in some instrumentation in the kernel that lets me track process
system calls and arguments and record a history of what's happened for later
analysis. Using this tool I can see what system calls were made before the
crash. You can think of it as a very simple strace, implemented in the kernel
-- so it's much faster.

I've noticed that before the segmentation fault, one worker thread closes a
file descriptor that is opened by another worker. I would call these extra
closes because the worker thread is closing files that it hasn't opened during
this connection. Based on the trace it looks like these extra close calls are
happening in apr_pool_clear/run_cleanups.

The thread that was using that file or socket will then make system calls with
bad (corrupted?) file descriptors. For example: sys_poll will be called with a
file descriptor that is already closed. The segfault often happens soon after
this -- presumably when something in the file structure is dereference.

Is it possible that worker threads are closing files that are opened by other
workers? I've skimmed through the Apache MPM worker source and can't see how
this could happen.
Comment 2 Paul Querna 2006-11-26 18:23:50 UTC 
Can you run and post a comment with the results of
'/usr/local/apache2_worker/bin/httpd -M' which lists all modules (including
dynamic modules)?
Comment 3 Jesse Pool 2006-11-26 19:53:17 UTC 
(In reply to comment #2)
> Can you run and post a comment with the results of
> '/usr/local/apache2_worker/bin/httpd -M' which lists all modules (including
> dynamic modules)?

Sure thing:

[root@rocksteady lcuser]$ /usr/local/apache2_worker/bin/httpd -M
Loaded Modules:
 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 log_config_module (static)
 env_module (static)
 setenvif_module (static)
 mpm_worker_module (static)
 http_module (static)
 mime_module (static)
 status_module (static)
 autoindex_module (static)
 asis_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 userdir_module (static)
 alias_module (static)
 so_module (static)
Syntax OK
Comment 4 Ruediger Pluem 2006-11-27 14:45:29 UTC 
Please do NEVER change the assignment of bugs.
Comment 5 Luca Saiu 2006-12-18 08:42:21 UTC 
Same for me, with essentially the same setup (SMP instead of multicore), and
using apache benchmark (ab).
Comment 6 Nadav Pe'er 2007-05-29 23:18:45 UTC 
I've got a similar problem, using WebLoad
(Dynamic modules disabled at compile time)

/usr/local/pkg/httpd-2.0.58/bin/httpd -l
Compiled in modules:
  core.c
  mod_perl.c
  mod_access.c
  mod_auth.c
  mod_cache.c
  mod_disk_cache.c
  mod_mem_cache.c
  util_ldap.c
  mod_auth_ldap.c
  mod_include.c
  mod_log_config.c
  mod_env.c
  mod_expires.c
  mod_setenvif.c
  mod_proxy.c
  proxy_connect.c
  proxy_ftp.c
  proxy_http.c
  mod_ssl.c
  worker.c
  http_core.c
  mod_mime.c
  mod_status.c
  mod_cgid.c
  mod_negotiation.c
  mod_dir.c
  mod_imap.c
  mod_actions.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c
Comment 7 Aleksey Midenkov 2007-11-27 07:15:57 UTC 
Please, install httpd, apr and apr-util debug info and make stack trace 
again...
Comment 8 Guilherme Assad 2008-06-03 11:28:56 UTC 
I have the same problem with apache 2.2.8 using siege to stress test my environment. When I reached 15000 transactions/second, approximately, apache started to refuse connections and process died with segmentation fault. 

At the error log I found:

*** glibc detected *** /opt/apache-2.2.8/bin/httpd: free(): invalid pointer: 0x75d00048 ***
======= Backtrace: =========
/lib/libc.so.6[0xf7d57911]
/lib/libc.so.6(__libc_free+0x84)[0xf7d58f84]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x160)[0xf7e7d9f0]
/opt/apache-2.2.8/bin/httpd[0x80896a9]
/opt/apache-2.2.8/lib/libapr-1.so.0[0xf7e7d14d]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x3d)[0xf7e7d8cd]
/opt/apache-2.2.8/bin/httpd[0x8087114]
/opt/apache-2.2.8/bin/httpd[0x80873e0]
/opt/apache-2.2.8/bin/httpd[0x80875c0]
/opt/apache-2.2.8/bin/httpd(ap_mpm_run+0x878)[0x8088388]
/opt/apache-2.2.8/bin/httpd(main+0x845)[0x8062135]
/lib/libc.so.6(__libc_start_main+0xdc)[0xf7d0987c]
/opt/apache-2.2.8/bin/httpd(apr_bucket_mmap_make+0x65)[0x8061341]
======= Memory map: ========
08048000-0809d000 r-xp 00000000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
0809d000-080a1000 rwxp 00054000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
080a1000-08581000 rwxp 080a1000 00:00 0                                  [heap]
75b00000-75b21000 rwxp 75b00000 00:00 0
75b21000-75c00000 ---p 75b21000 00:00 0
75c00000-75c21000 rwxp 75c00000 00:00 0
75c21000-75d00000 ---p 75c21000 00:00 0
75d00000-75d89000 rwxp 75d00000 00:00 0
75d89000-75e00000 ---p 75d89000 00:00 0
75e00000-75e21000 rwxp 75e00000 00:00 0
75e21000-75f00000 ---p 75e21000 00:00 0
75f00000-75f21000 rwxp 75f00000 00:00 0
75f21000-76000000 ---p 75f21000 00:00 0
76079000-76083000 r-xp 00000000 08:05 154940                             /lib/libgcc_s.so.1
76083000-76084000 rwxp 00009000 08:05 154940                             /lib/libgcc_s.so.1
7688e000-7688f000 ---p 7688e000 00:00 0
7688f000-7708f000 rwxp 7688f000 00:00 0
7708f000-77090000 ---p 7708f000 00:00 0
77090000-77890000 rwxp 77090000 00:00 0
77890000-77891000 ---p 77890000 00:00 0
77891000-78091000 rwxp 77891000 00:00 0
78091000-78092000 ---p 78091000 00:00 0
78092000-78892000 rwxp 78092000 00:00 0
f718f000-f7c80000 rwxs 00000000 00:08 213061645                          /dev/zero (deleted)
f7c80000-f7c88000 r-xp 00000000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c88000-f7c8a000 rwxp 00007000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c8a000-f7c9b000 r-xp 00000000 08:05 154903                             /lib/libnsl-2.4.so
f7c9b000-f7c9d000 rwxp 00010000 08:05 154903                             /lib/libnsl-2.4.so
f7c9d000-f7c9f000 rwxp f7c9d000 00:00 0
f7c9f000-f7ca5000 r-xp 00000000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca5000-f7ca7000 rwxp 00005000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca7000-f7caf000 r-xp 00000000 08:05 154909                             /lib/libnss_files-2.4.so
f7caf000-f7cb1000 rwxp 00007000 08:05 154909                             /lib/libnss_files-2.4.so
f7cba000-f7cc8000 r-xp 00000000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc8000-f7cc9000 rwxp 0000d000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc9000-f7ccc000 r-xp 00000000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccc000-f7ccd000 rwxp 00002000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccd000-f7ccf000 r-xp 00000000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7ccf000-f7cd0000 rwxp 00001000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7cd0000-f7cd4000 r-xp 00000000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd4000-f7cd5000 rwxp 00003000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd5000-f7cd9000 r-xp 00000000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cd9000-f7cda000 rwxp 00004000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cda000-f7cdc000 r-xp 00000000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdc000-f7cdd000 rwxp 00001000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdd000-f7ce0000 r-xp 00000000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce0000-f7ce1000 rwxp 00003000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce1000-f7ce3000 r-xp 00000000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce3000-f7ce4000 rwxp 00001000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce4000-f7ce9000 r-xp 00000000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7ce9000-f7cea000 rwxp 00004000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7cea000-f7cf1000 r-xp 00000000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf1000-f7cf2000 rwxp 00006000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf2000-f7cf4000 rwxp f7cf2000 00:00 0
f7cf4000-f7e0d000 r-xp 00000000 08:05 154892                             /lib/libc-2.4.so
f7e0d000-f7e0f000 r-xp 00118000 08:05 154892                             /lib/libc-2.4.so
f7e0f000-f7e11000 rwxp 0011a000 08:05 154892                             /lib/libc-2.4.so
f7e11000-f7e14000 rwxp f7e11000 00:00 0
f7e14000-f7e16000 r-xp 00000000 08:05 154898                             /lib/libdl-2.4.so
f7e16000-f7e18000 rwxp 00001000 08:05 154898                             /lib/libdl-2.4.so
f7e18000-f7e28000 r-xp 00000000 08:05 154918                             /lib/libpthread-2.4.so
f7e28000-f7e2a000 rwxp 0000f000 08:05 154918                             /lib/libpthread-2.4.so
f7e2a000-f7e2c000 rwxp f7e2a000 00:00 0
f7e2c000-f7e35000 r-xp 00000000 08:05 154896                             /lib/libcrypt-2.4.so
f7e35000-f7e38000 rwxp 00008000 08:05 154896                             /lib/libcrypt-2.4.so
f7e38000-f7e5f000 rwxp f7e38000 00:00 0
f7e5f000-f7e66000 r-xp 00000000 08:05 154922                             /lib/librt-2.4.so
f7e66000-f7e68000 rwxp 00006000 08:05 154922                             /lib/librt-2.4.so
f7e68000-f7e69000 rwxp f7e68000 00:00 0
f7e69000-f7e8b000 r-xp 00000000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8b000-f7e8c000 rwxp 00022000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8c000-f7ea9000 r-xp 00000000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7ea9000-f7eab000 rwxp 0001d000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7eab000-f7ebf000 r-xp 00000000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ebf000-f7ec0000 rwxp 00013000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ec0000-f7ee3000 r-xp 00000000 08:05 154900                             /lib/libm-2.4.so
f7ee3000-f7ee5000 rwxp 00022000 08:05 154900                             /lib/libm-2.4.so
f7ee5000-f7eea000 r-xp 00000000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eea000-f7eeb000 rwxp 00004000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eeb000-f7eed000 r-xp 00000000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eed000-f7eee000 rwxp 00001000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eee000-f7eef000 rwxp f7eee000 00:00 0
f7eef000-f7f09000 r-xp 00000000 08:05 154885                             /lib/ld-2.4.so
f7f09000-f7f0b000 rwxp 00019000 08:05 154885                             /lib/ld-2.4.so
ffff7000-ffffe000 rw-p ffff7000 00:00 0                                  [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
[Tue Jun 03 14:46:42 2008] [notice] child pid 1987 exit signal Aborted (6)
[Tue Jun 03 14:46:47 2008] [notice] child pid 4370 exit signal Segmentation fault (11)
*** glibc detected *** /opt/apache-2.2.8/bin/httpd: double free or corruption (out): 0x08141098 ***
======= Backtrace: =========
/lib/libc.so.6[0xf7d57911]
/lib/libc.so.6(__libc_free+0x84)[0xf7d58f84]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_allocator_destroy+0x2d)[0xf7e7d2ed]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x1ad)[0xf7e7da3d]
/opt/apache-2.2.8/bin/httpd[0x80896a9]
/opt/apache-2.2.8/lib/libapr-1.so.0[0xf7e7d14d]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x3d)[0xf7e7d8cd]
/opt/apache-2.2.8/bin/httpd[0x8087114]
/opt/apache-2.2.8/bin/httpd[0x80873e0]
/opt/apache-2.2.8/bin/httpd[0x80875c0]
/opt/apache-2.2.8/bin/httpd(ap_mpm_run+0x878)[0x8088388]
/opt/apache-2.2.8/bin/httpd(main+0x845)[0x8062135]
/lib/libc.so.6(__libc_start_main+0xdc)[0xf7d0987c]
/opt/apache-2.2.8/bin/httpd(apr_bucket_mmap_make+0x65)[0x8061341]
======= Memory map: ========
08048000-0809d000 r-xp 00000000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
0809d000-080a1000 rwxp 00054000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
080a1000-085f3000 rwxp 080a1000 00:00 0                                  [heap]
75d00000-75d37000 rwxp 75d00000 00:00 0
75d37000-75e00000 ---p 75d37000 00:00 0
75e00000-75e65000 rwxp 75e00000 00:00 0
75e65000-75f00000 ---p 75e65000 00:00 0
75f00000-75f7e000 rwxp 75f00000 00:00 0
75f7e000-76000000 ---p 75f7e000 00:00 0
76079000-76083000 r-xp 00000000 08:05 154940                             /lib/libgcc_s.so.1
76083000-76084000 rwxp 00009000 08:05 154940                             /lib/libgcc_s.so.1
7688e000-7688f000 ---p 7688e000 00:00 0
7688f000-7708f000 rwxp 7688f000 00:00 0
7708f000-77090000 ---p 7708f000 00:00 0
77090000-77890000 rwxp 77090000 00:00 0
77890000-77891000 ---p 77890000 00:00 0
77891000-78091000 rwxp 77891000 00:00 0
78091000-78092000 ---p 78091000 00:00 0
78092000-78892000 rwxp 78092000 00:00 0
f718f000-f7c80000 rwxs 00000000 00:08 213061645                          /dev/zero (deleted)
f7c80000-f7c88000 r-xp 00000000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c88000-f7c8a000 rwxp 00007000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c8a000-f7c9b000 r-xp 00000000 08:05 154903                             /lib/libnsl-2.4.so
f7c9b000-f7c9d000 rwxp 00010000 08:05 154903                             /lib/libnsl-2.4.so
f7c9d000-f7c9f000 rwxp f7c9d000 00:00 0
f7c9f000-f7ca5000 r-xp 00000000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca5000-f7ca7000 rwxp 00005000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca7000-f7caf000 r-xp 00000000 08:05 154909                             /lib/libnss_files-2.4.so
f7caf000-f7cb1000 rwxp 00007000 08:05 154909                             /lib/libnss_files-2.4.so
f7cba000-f7cc8000 r-xp 00000000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc8000-f7cc9000 rwxp 0000d000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc9000-f7ccc000 r-xp 00000000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccc000-f7ccd000 rwxp 00002000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccd000-f7ccf000 r-xp 00000000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7ccf000-f7cd0000 rwxp 00001000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7cd0000-f7cd4000 r-xp 00000000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd4000-f7cd5000 rwxp 00003000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd5000-f7cd9000 r-xp 00000000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cd9000-f7cda000 rwxp 00004000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cda000-f7cdc000 r-xp 00000000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdc000-f7cdd000 rwxp 00001000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdd000-f7ce0000 r-xp 00000000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce0000-f7ce1000 rwxp 00003000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce1000-f7ce3000 r-xp 00000000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce3000-f7ce4000 rwxp 00001000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce4000-f7ce9000 r-xp 00000000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7ce9000-f7cea000 rwxp 00004000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7cea000-f7cf1000 r-xp 00000000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf1000-f7cf2000 rwxp 00006000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf2000-f7cf4000 rwxp f7cf2000 00:00 0
f7cf4000-f7e0d000 r-xp 00000000 08:05 154892                             /lib/libc-2.4.so
f7e0d000-f7e0f000 r-xp 00118000 08:05 154892                             /lib/libc-2.4.so
f7e0f000-f7e11000 rwxp 0011a000 08:05 154892                             /lib/libc-2.4.so
f7e11000-f7e14000 rwxp f7e11000 00:00 0
f7e14000-f7e16000 r-xp 00000000 08:05 154898                             /lib/libdl-2.4.so
f7e16000-f7e18000 rwxp 00001000 08:05 154898                             /lib/libdl-2.4.so
f7e18000-f7e28000 r-xp 00000000 08:05 154918                             /lib/libpthread-2.4.so
f7e28000-f7e2a000 rwxp 0000f000 08:05 154918                             /lib/libpthread-2.4.so
f7e2a000-f7e2c000 rwxp f7e2a000 00:00 0
f7e2c000-f7e35000 r-xp 00000000 08:05 154896                             /lib/libcrypt-2.4.so
f7e35000-f7e38000 rwxp 00008000 08:05 154896                             /lib/libcrypt-2.4.so
f7e38000-f7e5f000 rwxp f7e38000 00:00 0
f7e5f000-f7e66000 r-xp 00000000 08:05 154922                             /lib/librt-2.4.so
f7e66000-f7e68000 rwxp 00006000 08:05 154922                             /lib/librt-2.4.so
f7e68000-f7e69000 rwxp f7e68000 00:00 0
f7e69000-f7e8b000 r-xp 00000000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8b000-f7e8c000 rwxp 00022000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8c000-f7ea9000 r-xp 00000000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7ea9000-f7eab000 rwxp 0001d000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7eab000-f7ebf000 r-xp 00000000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ebf000-f7ec0000 rwxp 00013000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ec0000-f7ee3000 r-xp 00000000 08:05 154900                             /lib/libm-2.4.so
f7ee3000-f7ee5000 rwxp 00022000 08:05 154900                             /lib/libm-2.4.so
f7ee5000-f7eea000 r-xp 00000000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eea000-f7eeb000 rwxp 00004000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eeb000-f7eed000 r-xp 00000000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eed000-f7eee000 rwxp 00001000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eee000-f7eef000 rwxp f7eee000 00:00 0
f7eef000-f7f09000 r-xp 00000000 08:05 154885                             /lib/ld-2.4.so
f7f09000-f7f0b000 rwxp 00019000 08:05 154885                             /lib/ld-2.4.so
ffff7000-ffffe000 rw-p ffff7000 00:00 0                                  [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
[Tue Jun 03 14:47:30 2008] [notice] child pid 23953 exit signal Aborted (6)
[Tue Jun 03 14:47:34 2008] [notice] child pid 25768 exit signal Segmentation fault (11)
[Tue Jun 03 14:47:39 2008] [notice] child pid 28099 exit signal Segmentation fault (11)
*** glibc detected *** /opt/apache-2.2.8/bin/httpd: double free or corruption (out): 0x08141098 ***
======= Backtrace: =========
/lib/libc.so.6[0xf7d57911]
/lib/libc.so.6(__libc_free+0x84)[0xf7d58f84]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x160)[0xf7e7d9f0]
/opt/apache-2.2.8/bin/httpd[0x80896a9]
/opt/apache-2.2.8/lib/libapr-1.so.0[0xf7e7d14d]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x3d)[0xf7e7d8cd]
/opt/apache-2.2.8/bin/httpd[0x8087114]
/opt/apache-2.2.8/bin/httpd[0x80873e0]
/opt/apache-2.2.8/bin/httpd[0x80875c0]
/opt/apache-2.2.8/bin/httpd(ap_mpm_run+0x878)[0x8088388]
/opt/apache-2.2.8/bin/httpd(main+0x845)[0x8062135]
/lib/libc.so.6(__libc_start_main+0xdc)[0xf7d0987c]
/opt/apache-2.2.8/bin/httpd(apr_bucket_mmap_make+0x65)[0x8061341]
======= Memory map: ========
08048000-0809d000 r-xp 00000000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
0809d000-080a1000 rwxp 00054000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
080a1000-0859d000 rwxp 080a1000 00:00 0                                  [heap]
75900000-75935000 rwxp 75900000 00:00 0
75935000-75a00000 ---p 75935000 00:00 0
75b00000-75b3d000 rwxp 75b00000 00:00 0
75b3d000-75c00000 ---p 75b3d000 00:00 0
75c00000-75c31000 rwxp 75c00000 00:00 0
75c31000-75d00000 ---p 75c31000 00:00 0
75d00000-75d23000 rwxp 75d00000 00:00 0
75d23000-75e00000 ---p 75d23000 00:00 0
75e00000-75e39000 rwxp 75e00000 00:00 0
75e39000-75f00000 ---p 75e39000 00:00 0
75f00000-75f45000 rwxp 75f00000 00:00 0
75f45000-76000000 ---p 75f45000 00:00 0
76079000-76083000 r-xp 00000000 08:05 154940                             /lib/libgcc_s.so.1
76083000-76084000 rwxp 00009000 08:05 154940                             /lib/libgcc_s.so.1
7688e000-7688f000 ---p 7688e000 00:00 0
7688f000-7708f000 rwxp 7688f000 00:00 0
7708f000-77090000 ---p 7708f000 00:00 0
77090000-77890000 rwxp 77090000 00:00 0
77890000-77891000 ---p 77890000 00:00 0
77891000-78091000 rwxp 77891000 00:00 0
78091000-78092000 ---p 78091000 00:00 0
78092000-78892000 rwxp 78092000 00:00 0
f718f000-f7c80000 rwxs 00000000 00:08 213061645                          /dev/zero (deleted)
f7c80000-f7c88000 r-xp 00000000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c88000-f7c8a000 rwxp 00007000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c8a000-f7c9b000 r-xp 00000000 08:05 154903                             /lib/libnsl-2.4.so
f7c9b000-f7c9d000 rwxp 00010000 08:05 154903                             /lib/libnsl-2.4.so
f7c9d000-f7c9f000 rwxp f7c9d000 00:00 0
f7c9f000-f7ca5000 r-xp 00000000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca5000-f7ca7000 rwxp 00005000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca7000-f7caf000 r-xp 00000000 08:05 154909                             /lib/libnss_files-2.4.so
f7caf000-f7cb1000 rwxp 00007000 08:05 154909                             /lib/libnss_files-2.4.so
f7cba000-f7cc8000 r-xp 00000000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc8000-f7cc9000 rwxp 0000d000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc9000-f7ccc000 r-xp 00000000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccc000-f7ccd000 rwxp 00002000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccd000-f7ccf000 r-xp 00000000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7ccf000-f7cd0000 rwxp 00001000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7cd0000-f7cd4000 r-xp 00000000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd4000-f7cd5000 rwxp 00003000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd5000-f7cd9000 r-xp 00000000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cd9000-f7cda000 rwxp 00004000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cda000-f7cdc000 r-xp 00000000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdc000-f7cdd000 rwxp 00001000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdd000-f7ce0000 r-xp 00000000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce0000-f7ce1000 rwxp 00003000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce1000-f7ce3000 r-xp 00000000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce3000-f7ce4000 rwxp 00001000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce4000-f7ce9000 r-xp 00000000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7ce9000-f7cea000 rwxp 00004000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7cea000-f7cf1000 r-xp 00000000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf1000-f7cf2000 rwxp 00006000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf2000-f7cf4000 rwxp f7cf2000 00:00 0
f7cf4000-f7e0d000 r-xp 00000000 08:05 154892                             /lib/libc-2.4.so
f7e0d000-f7e0f000 r-xp 00118000 08:05 154892                             /lib/libc-2.4.so
f7e0f000-f7e11000 rwxp 0011a000 08:05 154892                             /lib/libc-2.4.so
f7e11000-f7e14000 rwxp f7e11000 00:00 0
f7e14000-f7e16000 r-xp 00000000 08:05 154898                             /lib/libdl-2.4.so
f7e16000-f7e18000 rwxp 00001000 08:05 154898                             /lib/libdl-2.4.so
f7e18000-f7e28000 r-xp 00000000 08:05 154918                             /lib/libpthread-2.4.so
f7e28000-f7e2a000 rwxp 0000f000 08:05 154918                             /lib/libpthread-2.4.so
f7e2a000-f7e2c000 rwxp f7e2a000 00:00 0
f7e2c000-f7e35000 r-xp 00000000 08:05 154896                             /lib/libcrypt-2.4.so
f7e35000-f7e38000 rwxp 00008000 08:05 154896                             /lib/libcrypt-2.4.so
f7e38000-f7e5f000 rwxp f7e38000 00:00 0
f7e5f000-f7e66000 r-xp 00000000 08:05 154922                             /lib/librt-2.4.so
f7e66000-f7e68000 rwxp 00006000 08:05 154922                             /lib/librt-2.4.so
f7e68000-f7e69000 rwxp f7e68000 00:00 0
f7e69000-f7e8b000 r-xp 00000000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8b000-f7e8c000 rwxp 00022000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8c000-f7ea9000 r-xp 00000000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7ea9000-f7eab000 rwxp 0001d000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7eab000-f7ebf000 r-xp 00000000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ebf000-f7ec0000 rwxp 00013000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ec0000-f7ee3000 r-xp 00000000 08:05 154900                             /lib/libm-2.4.so
f7ee3000-f7ee5000 rwxp 00022000 08:05 154900                             /lib/libm-2.4.so
f7ee5000-f7eea000 r-xp 00000000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eea000-f7eeb000 rwxp 00004000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eeb000-f7eed000 r-xp 00000000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eed000-f7eee000 rwxp 00001000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eee000-f7eef000 rwxp f7eee000 00:00 0
f7eef000-f7f09000 r-xp 00000000 08:05 154885                             /lib/ld-2.4.so
f7f09000-f7f0b000 rwxp 00019000 08:05 154885                             /lib/ld-2.4.so
ffff7000-ffffe000 rw-p ffff7000 00:00 0                                  [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
[Tue Jun 03 14:48:37 2008] [notice] child pid 22144 exit signal Aborted (6)
[Tue Jun 03 14:48:54 2008] [error] [client 10.6.12.243] request failed: error reading the headers
[Tue Jun 03 14:48:55 2008] [notice] child pid 30329 exit signal Segmentation fault (11)
[Tue Jun 03 14:50:08 2008] [notice] child pid 30977 exit signal Segmentation fault (11)
[Tue Jun 03 14:50:15 2008] [notice] child pid 2153 exit signal Segmentation fault (11)
[Tue Jun 03 14:50:43 2008] [notice] child pid 14969 exit signal Segmentation fault (11)
[Tue Jun 03 14:51:51 2008] [notice] child pid 13548 exit signal Segmentation fault (11)
[Tue Jun 03 14:52:11 2008] [notice] child pid 22652 exit signal Segmentation fault (11)
*** glibc detected *** /opt/apache-2.2.8/bin/httpd: double free or corruption (out): 0x75f00048 ***
======= Backtrace: =========
/lib/libc.so.6[0xf7d57911]
/lib/libc.so.6(__libc_free+0x84)[0xf7d58f84]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x160)[0xf7e7d9f0]
/opt/apache-2.2.8/bin/httpd[0x80896a9]
/opt/apache-2.2.8/lib/libapr-1.so.0[0xf7e7d14d]
/opt/apache-2.2.8/lib/libapr-1.so.0(apr_pool_destroy+0x3d)[0xf7e7d8cd]
/opt/apache-2.2.8/bin/httpd[0x8087114]
/opt/apache-2.2.8/bin/httpd[0x80873e0]
/opt/apache-2.2.8/bin/httpd[0x80875c0]
/opt/apache-2.2.8/bin/httpd(ap_mpm_run+0x878)[0x8088388]
/opt/apache-2.2.8/bin/httpd(main+0x845)[0x8062135]
/lib/libc.so.6(__libc_start_main+0xdc)[0xf7d0987c]
/opt/apache-2.2.8/bin/httpd(apr_bucket_mmap_make+0x65)[0x8061341]
======= Memory map: ========
08048000-0809d000 r-xp 00000000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
0809d000-080a1000 rwxp 00054000 08:0a 624011                             /opt/apache-2.2.8/bin/httpd
080a1000-0851a000 rwxp 080a1000 00:00 0                                  [heap]
75500000-75539000 rwxp 75500000 00:00 0
75539000-75600000 ---p 75539000 00:00 0
75700000-7577b000 rwxp 75700000 00:00 0
7577b000-75800000 ---p 7577b000 00:00 0
75800000-758a1000 rwxp 75800000 00:00 0
758a1000-75900000 ---p 758a1000 00:00 0
75900000-7597f000 rwxp 75900000 00:00 0
7597f000-75a00000 ---p 7597f000 00:00 0
75a00000-75a70000 rwxp 75a00000 00:00 0
75a70000-75b00000 ---p 75a70000 00:00 0
75b00000-75b72000 rwxp 75b00000 00:00 0
75b72000-75c00000 ---p 75b72000 00:00 0
75c00000-75c59000 rwxp 75c00000 00:00 0
75c59000-75d00000 ---p 75c59000 00:00 0
75d00000-75d37000 rwxp 75d00000 00:00 0
75d37000-75e00000 ---p 75d37000 00:00 0
75e00000-75e6b000 rwxp 75e00000 00:00 0
75e6b000-75f00000 ---p 75e6b000 00:00 0
75f00000-76000000 rwxp 75f00000 00:00 0
76079000-76083000 r-xp 00000000 08:05 154940                             /lib/libgcc_s.so.1
76083000-76084000 rwxp 00009000 08:05 154940                             /lib/libgcc_s.so.1
7688e000-7688f000 ---p 7688e000 00:00 0
7688f000-7708f000 rwxp 7688f000 00:00 0
7708f000-77090000 ---p 7708f000 00:00 0
77090000-77890000 rwxp 77090000 00:00 0
77890000-77891000 ---p 77890000 00:00 0
77891000-78091000 rwxp 77891000 00:00 0
78091000-78092000 ---p 78091000 00:00 0
78092000-78892000 rwxp 78092000 00:00 0
f718f000-f7c80000 rwxs 00000000 00:08 213061645                          /dev/zero (deleted)
f7c80000-f7c88000 r-xp 00000000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c88000-f7c8a000 rwxp 00007000 08:05 154913                             /lib/libnss_nis-2.4.so
f7c8a000-f7c9b000 r-xp 00000000 08:05 154903                             /lib/libnsl-2.4.so
f7c9b000-f7c9d000 rwxp 00010000 08:05 154903                             /lib/libnsl-2.4.so
f7c9d000-f7c9f000 rwxp f7c9d000 00:00 0
f7c9f000-f7ca5000 r-xp 00000000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca5000-f7ca7000 rwxp 00005000 08:05 154905                             /lib/libnss_compat-2.4.so
f7ca7000-f7caf000 r-xp 00000000 08:05 154909                             /lib/libnss_files-2.4.so
f7caf000-f7cb1000 rwxp 00007000 08:05 154909                             /lib/libnss_files-2.4.so
f7cba000-f7cc8000 r-xp 00000000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc8000-f7cc9000 rwxp 0000d000 08:0a 624342                             /opt/apache-2.2.8/modules/mod_rewrite.so
f7cc9000-f7ccc000 r-xp 00000000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccc000-f7ccd000 rwxp 00002000 08:0a 624326                             /opt/apache-2.2.8/modules/mod_imagemap.so
f7ccd000-f7ccf000 r-xp 00000000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7ccf000-f7cd0000 rwxp 00001000 08:0a 624351                             /opt/apache-2.2.8/modules/mod_vhost_alias.so
f7cd0000-f7cd4000 r-xp 00000000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd4000-f7cd5000 rwxp 00003000 08:0a 624328                             /opt/apache-2.2.8/modules/mod_info.so
f7cd5000-f7cd9000 r-xp 00000000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cd9000-f7cda000 rwxp 00004000 08:0a 624345                             /opt/apache-2.2.8/modules/mod_status.so
f7cda000-f7cdc000 r-xp 00000000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdc000-f7cdd000 rwxp 00001000 08:0a 624343                             /opt/apache-2.2.8/modules/mod_setenvif.so
f7cdd000-f7ce0000 r-xp 00000000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce0000-f7ce1000 rwxp 00003000 08:0a 624333                             /opt/apache-2.2.8/modules/mod_mime.so
f7ce1000-f7ce3000 r-xp 00000000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce3000-f7ce4000 rwxp 00001000 08:0a 624320                             /opt/apache-2.2.8/modules/mod_expires.so
f7ce4000-f7ce9000 r-xp 00000000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7ce9000-f7cea000 rwxp 00004000 08:0a 624329                             /opt/apache-2.2.8/modules/mod_log_config.so
f7cea000-f7cf1000 r-xp 00000000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf1000-f7cf2000 rwxp 00006000 08:0a 624307                             /opt/apache-2.2.8/modules/mod_cache.so
f7cf2000-f7cf4000 rwxp f7cf2000 00:00 0
f7cf4000-f7e0d000 r-xp 00000000 08:05 154892                             /lib/libc-2.4.so
f7e0d000-f7e0f000 r-xp 00118000 08:05 154892                             /lib/libc-2.4.so
f7e0f000-f7e11000 rwxp 0011a000 08:05 154892                             /lib/libc-2.4.so
f7e11000-f7e14000 rwxp f7e11000 00:00 0
f7e14000-f7e16000 r-xp 00000000 08:05 154898                             /lib/libdl-2.4.so
f7e16000-f7e18000 rwxp 00001000 08:05 154898                             /lib/libdl-2.4.so
f7e18000-f7e28000 r-xp 00000000 08:05 154918                             /lib/libpthread-2.4.so
f7e28000-f7e2a000 rwxp 0000f000 08:05 154918                             /lib/libpthread-2.4.so
f7e2a000-f7e2c000 rwxp f7e2a000 00:00 0
f7e2c000-f7e35000 r-xp 00000000 08:05 154896                             /lib/libcrypt-2.4.so
f7e35000-f7e38000 rwxp 00008000 08:05 154896                             /lib/libcrypt-2.4.so
f7e38000-f7e5f000 rwxp f7e38000 00:00 0
f7e5f000-f7e66000 r-xp 00000000 08:05 154922                             /lib/librt-2.4.so
f7e66000-f7e68000 rwxp 00006000 08:05 154922                             /lib/librt-2.4.so
f7e68000-f7e69000 rwxp f7e68000 00:00 0
f7e69000-f7e8b000 r-xp 00000000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8b000-f7e8c000 rwxp 00022000 08:0a 624262                             /opt/apache-2.2.8/lib/libapr-1.so.0.2.12
f7e8c000-f7ea9000 r-xp 00000000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7ea9000-f7eab000 rwxp 0001d000 08:08 64848                              /usr/lib/libexpat.so.1.5.0
f7eab000-f7ebf000 r-xp 00000000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ebf000-f7ec0000 rwxp 00013000 08:0a 624261                             /opt/apache-2.2.8/lib/libaprutil-1.so.0.2.12
f7ec0000-f7ee3000 r-xp 00000000 08:05 154900                             /lib/libm-2.4.so
f7ee3000-f7ee5000 rwxp 00022000 08:05 154900                             /lib/libm-2.4.so
f7ee5000-f7eea000 r-xp 00000000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eea000-f7eeb000 rwxp 00004000 08:0a 624332                             /opt/apache-2.2.8/modules/mod_mem_cache.so
f7eeb000-f7eed000 r-xp 00000000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eed000-f7eee000 rwxp 00001000 08:0a 624303                             /opt/apache-2.2.8/modules/mod_authz_host.so
f7eee000-f7eef000 rwxp f7eee000 00:00 0
f7eef000-f7f09000 r-xp 00000000 08:05 154885                             /lib/ld-2.4.so
f7f09000-f7f0b000 rwxp 00019000 08:05 154885                             /lib/ld-2.4.so
ffff7000-ffffe000 rw-p ffff7000 00:00 0                                  [stack]
ffffe000-fffff000 r-xp ffffe000 00:00 0
[Tue Jun 03 14:52:39 2008] [notice] child pid 2521 exit signal Aborted (6)
[Tue Jun 03 14:52:56 2008] [notice] child pid 10313 exit signal Segmentation fault (11)
[Tue Jun 03 14:53:59 2008] [notice] child pid 7088 exit signal Segmentation fault (11)
[Tue Jun 03 14:54:15 2008] [notice] child pid 14367 exit signal Segmentation fault (11)


My apache is running with this modules:
 ./httpd -M
Loaded Modules:
 core_module (static)
 mpm_worker_module (static)
 http_module (static)
 so_module (static)
 authz_host_module (shared)
 cache_module (shared)
 mem_cache_module (shared)
 log_config_module (shared)
 expires_module (shared)
 mime_module (shared)
 setenvif_module (shared)
 status_module (shared)
 info_module (shared)
 vhost_alias_module (shared)
 imagemap_module (shared)
 rewrite_module (shared)
Syntax OK

And the main performance configuration is set this way:
ServerRoot "/opt/apache"
LockFile logs/accept.lock
PidFile logs/httpd.pid

Timeout                 20
KeepAlive               On
MaxKeepAliveRequests    200
KeepAliveTimeout        3

ThreadLimit             256
ThreadsPerChild         256
StartServers            5
ServerLimit             200
MinSpareThreads         512
MaxSpareThreads         1024
MaxRequestsPerChild     10000

ListenBacklog           1024


User nobody
Group nobody
UseCanonicalName Off
HostnameLookups Off
ServerSignature Off

EnableMMAP off
EnableSendFile off
Comment 9 Ruediger Pluem 2008-06-03 13:29:30 UTC 
Have you checked (In reply to comment #8)
> I have the same problem with apache 2.2.8 using siege to stress test my
> environment. When I reached 15000 transactions/second, approximately, apache
> started to refuse connections and process died with segmentation fault. 
> 

Have you checked if you are hit by PR44402? Can you please try http://svn.apache.org/viewvc?view=rev&revision=631362?
Comment 10 Jesse Pool 2008-06-03 14:48:35 UTC 
Yep. I haven't tried the patch to confirm, but that looks like the same issue. Back when I filed this bug, I quickly got around the problem by doing something like this:

void ap_push_pool(fd_queue_info_t * queue_info,
                                    apr_pool_t * pool_to_recycle)
{
    /* If we have been given a pool to recycle, atomically link
     * it into the queue_info's list of recycled pools
     */
    if (pool_to_recycle) {
#if 0
        struct recycled_pool *new_recycle;
        new_recycle = (struct recycled_pool *) apr_palloc(pool_to_recycle,
                                                          sizeof
                                                          (*new_recycle));
        new_recycle->pool = pool_to_recycle;
        for (;;) {
            new_recycle->next = queue_info->recycled_pools;
            if (apr_atomic_casptr
                ((volatile void **) &(queue_info->recycled_pools),
                 new_recycle, new_recycle->next) == new_recycle->next) {
                break;
            }
        }
#else
        apr_pool_destroy(pool_to_recycle);
#endif
    }
}

...Sorry I didn't have time to investigate further at the time. If you're curious, that change gave 5% slowdown on my Webstone benchmark.
Comment 11 Guilherme Assad 2008-06-04 08:00:22 UTC 
(In reply to comment #9)
> Have you checked (In reply to comment #8)
> > I have the same problem with apache 2.2.8 using siege to stress test my
> > environment. When I reached 15000 transactions/second, approximately, apache
> > started to refuse connections and process died with segmentation fault. 
> > 
> 
> Have you checked if you are hit by PR44402? Can you please try
> http://svn.apache.org/viewvc?view=rev&revision=631362?
> 

It seems to be related to PR44402. I have some core files with the same pattern from the ones in that bug report.

I will try to patch the apache server with the code you sent.
Comment 12 William A. Rowe Jr. 2010-10-06 19:06:04 UTC 
As this appeared related to 44402, and no further feedback arrived, I'm going ahead and dupping out this report to that bug, which is long fixed.

*** This bug has been marked as a duplicate of bug 44402 ***