Bug 27550 - pcre symbol issues for apache modules
pcre symbol issues for apache modules
Status: RESOLVED FIXED
Product: Apache httpd-2
Classification: Unclassified
Component: Core
2.0-HEAD
All All
: P3 major with 12 votes (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
: PatchAvailable
: 23952 26088 28781 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2004-03-09 22:10 UTC by Andres Salomon
Modified: 2005-05-23 13:49 UTC (History)
10 users (show)



Attachments
patch to link against external libpcre (14.59 KB, patch)
2004-03-09 22:11 UTC, Andres Salomon
Details | Diff
patch to rename internal pcre functions (34.71 KB, patch)
2004-03-09 22:12 UTC, Andres Salomon
Details | Diff
Amended version of patch by Andres Salomon, with typo fixed by Paul Argentoff (14.59 KB, patch)
2004-11-10 09:16 UTC, Richard W.M. Jones
Details | Diff
more intelligent patch to link against PCRE (8.45 KB, patch)
2004-12-06 22:07 UTC, Andres Salomon
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andres Salomon 2004-03-09 22:10:05 UTC
Currently, apache2 includes a full version of pcre-3.9 in its sources;
this is built during compilation, and statically linked against.  The
symbols from this library, are exported to apache modules.  This is
undesirable for some apache modules; for example, php.  Php links
against libpcre on the system; however, it will happily pick up the pcre
symbols in the apache binary before the symbols in the pcre shared
library.  This breaks php functions such as pcre_match(), which expect a
relatively new version of pcre to be used (3.9 doesn't cut it; it always
fails w/ that).

There are a number of solutions, here.  I attempted upgrading the pcre
included in apache; however, I had no way to know if the various win32
hacks introduced to the pcre build system would still work.  I opted to
not bother, since I have no way to test such work except for on a unix
system.

An alternate solution is to check for pcre on the build system, and link
against that instead of building pcre for apache.  This works, but
introduces other issues.  With apache2 looking for the posix regex
functions in shared libraries, it ends up using the glibc
implementations for reg{comp,exec,free,error} instead of the pcre
implementations of these function.  On my system, the glibc
implementations segfault somewhere inside glibc; it's much safer to
stick w/ the pcre versions.  So, aside from the autoconf-ery that's
necessary to do the pcre system checks, there are some additional code
changes necessary to make sure that the pcre functions are called
instead of the glibc functions.  This is mainly some ugly code
duplication.  The patch that implements this change is here:
<http://sloth.voxel.net/~dilinger/020-external_pcre.patch>

A third solution is to simply change the internal pcre library to have
its own namespace.  This ensures that apache uses a pcre that it's been
fully tested with, while apache modules don't use them in place of a
system-supplied pcre.  What I've done in the following patch is to
prefix the posix regex functions in pcre with ap_pcreposix, and to
prefix the other pcre regex functions with ap_pcre.  The interface that
apache is _suppose to_ export to modules has not changed.  That patch is
here:
<http://sloth.voxel.net/~dilinger/021-pcre_mangle_symbols.patch>

I prefer the 021-pcre_mangle_symbols.patch patch, even though it's
larger; it ensures stability, at the cost of some code duplication.  The
020-external_pcre.patch patch isn't as clean as I'd hoped, and for the
added complexity, not much memory is saved.

Anyways, links to both patches are supplied so that you folks can decide
between the preferred method.  Please consider applying either one of my
patches, or an equivalent patch, so that pcre issues in external apache
modules are solved.
Comment 1 Andres Salomon 2004-03-09 22:11:47 UTC
Created attachment 10733 [details]
patch to link against external libpcre
Comment 2 Andres Salomon 2004-03-09 22:12:19 UTC
Created attachment 10734 [details]
patch to rename internal pcre functions
Comment 3 Joe Orton 2004-03-09 23:57:30 UTC
*** Bug 26088 has been marked as a duplicate of this bug. ***
Comment 4 OpenMacNews 2004-08-10 06:52:48 UTC
hi all,

this problem's been around for awhile ... for at least a couple of years!
	http://groups.google.com/groups?hl=en&lr=&ie=UTF
-8&scoring=d&q=pcre+apache+%22multiple+definitions+of+symbol%22&btnG=Search

it keeps getting reported primarily to the php boards, and repeatedly referred *back* as an Apache bug.

bottom line:
	
######################################################################
Andres' patch (http://nagoya.apache.org/bugzilla/showattachment.cgi?attach_id=10734) 
seems to fix the problem.  can this get integrated into HEAD & the 2.0.50 RELEASE?
######################################################################

on OSX, for quite awhile, i've been able to get around it by buildg httpd with its internal/included 
pcre39, building an external build of pcre39, and linking php against the external lib via the LDFLACGS 
& CPPFLAGS

however, with recent (for me, since yesterday)  httpd-head, building php --with-apxs aagain causes 
conflicts with apache's included pcre.  any inclusion/combination of any other (later) version, OR the 
inclusion of the php config param "--with-pcre-regex=..." results in 'make' errors/failures.

reading thru Andres' patch submission here, i tried the 'replace-namespace' patch (link above).  per 
Andres, it seems to do the trick for Debian ...

as for me, with patch applied in OSX, i'm now able to successfully build httpd w/ internal pcre -- now in 
its own namespace -- and build php500 --with-pcre against an external build of pcre45.

here _my_ 'recipe' for success ...

three 'pieces':
(1) pcre 45
(2) httpd 2.0.50 (there's a few other issues i'm having with latest HEAD, so i dropped back to 2.0.50 for 
this)
(3) php 500 release

also:
	% uname -v
		Darwin Kernel Version 7.5.0: Thu Aug  5 19:26:16 PDT 2004; root:xnu/xnu-517.7.21.obj~3/
RELEASE_PPC 
	% glibtool --version
		ltmain.sh (GNU libtool) 1.5.8 (1.1220.2.117 2004/08/04 14:12:05)
	% automake --version
		automake (GNU automake) 1.9
	% autoconf --version
		autoconf (GNU Autoconf) 2.59
	% gcc --version
		gcc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1666)
	% /usr/local/ssl/bin/openssl version
		OpenSSL 0.9.7d 17 Mar 2004


############################################################
pcre-4.5
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-4.5.tar.gz
gnutar zxf pcre-4.5.tar.gz
cd /usr/ports/pcre-4.5
unsetenv CFLAGS CPPFLAGS CXX CXXFLAGS LDFLAGS LDDLFLAGS LD_PREBIND LINGUAS LC_ALL LANG
glibtoolize --force --copy && aclocal && autoconf
./configure \
--prefix=/usr/local/pcre45 \
--enable-shared --disable-static
make
make install

############################################################
httpd-2.0.50

wget ftp://apache.secsup.org/pub/apache/dist/httpd/httpd-2.0.50.tar.gz
gnutar zxf httpd-2.0.50.tar.gz
cd /usr/ports/httpd-2.0.50

curl -f -L -o patch.pcre.txt "http://nagoya.apache.org/bugzilla/showattachment.cgi?attach_id=10734"
patch -p2 < patch.pcre.txt
unsetenv CFLAGS CPPFLAGS CXX CXXFLAGS LDFLAGS LDDLFLAGS LD_PREBIND LC_ALL LANG LINGUAS
./buildconf

# enable linking against dlcompat's libdl
=============================================================
(EDITOR) /usr/ports/httpd-2.0.50/srclib/apr/configure
@10433
---	enable_dlopen=no
+++	enable_dlopen=yes
	enable_win32_dll=no
=============================================================

./configure \
--with-mpm=worker \
--enable-layout=Darwin --with-port=80 \
--enable-mods-shared=all --enable-so --disable-static \
--sysconfdir=/etc/apache2 \
--enable-dav --enable-dav-fs --enable-dav-lock \
--enable-ssl --with-ssl=/usr/local/ssl \
--disable-suexec \
--with-z \
--enable-cgi \
--enable-proxy \
--enable-proxy-connect \
--enable-proxy-ftp \
--enable-proxy-http \
--enable-logio \
--enable-authn-dbm --enable-authz-dbm \
--with-imap


# these lib links won't pick up from cmd line ... do it manually for now
=============================================================
(EDITOR) /usr/ports/httpd-2.0.50/build/config_vars.mk
	EXTRA_CFLAGS = -g -O2
	EXTRA_CXXFLAGS =
---	EXTRA_LDFLAGS = -L/usr/local/lib  -L/usr/local/ssl/lib
+++	EXTRA_LDFLAGS = -bind_at_load -L/usr/local/lib  -L/usr/local/ssl/lib
---	EXTRA_LIBS = -lssl -lcrypto
+++	EXTRA_LIBS = -lssl -lcrypto -lz -ldl -lexpat
=============================================================

make
make install

#################################################################
php-5.0.0
wget http://us2.php.net/distributions/php-5.0.0.tar.gz
gnutar zxf php-5.0.0.tar.gz

unsetenv CFLAGS CPPFLAGS CXX CXXFLAGS LDFLAGS LDDLFLAGS LD_PREBIND EXTRA_LDFLAGS 
EXTRA_LIBS LC_ALL LANG LINGUAS ;\
setenv LC_ALL C ;\
setenv LANG C ;\
setenv CPPFLAGS "-I/usr/local/ssl/include -I/usr/local/pcre45/include -I/usr/local/include" ;\
setenv LDFLAGS "-bind_at_load  -L/usr/local/ssl/lib -lssl -lcrypto -L/usr/local/pcre45/lib -lpcre"

=============================================================
(EDITOR) /usr/ports/php-5.0.0/configure.in
---	AC_PROG_RANLIB
+++	AC_PROG_LIBTOOL
=============================================================

cd /usr/ports/php-5.0.0
autoconf
ranlib /usr/local/pgsql/lib/libpq.a

./configure \
--disable-debug \
--prefix=/usr --with-layout=PHP \
--with-config-file-path=/etc/php5 --sysconfdir=/etc/php5 \
--enable-shared --disable-static \
--libdir=/System/Library/PHP \
--includedir=/usr/include \
--mandir=/usr/local/man \
--localstatedir=/var/php4 \
--with-db4=/usr \
--with-pgsql=/usr/local/pgsql --without-mysql \
--enable-cli --with-pear=/System/Library/PHP \
--disable-cgi \
--with-apxs2=/usr/sbin/apxs \
--disable-dmalloc \
--with-tsrm-pthreads \
--enable-shmop --enable-sockets \
--enable-inline-optimization \
--enable-xml --enable-libxml --with-libxml-dir=/usr \
--with-java=/Library/Java/Home \
--with-openssl=/usr/local/ssl --with-openssl-dir=/usr/local/ssl \
--with-zlib --with-zlib-dir=/usr \
--with-imap=/usr/local/imap --enable-mailparse \
--with-imap-ssl=/usr/local/ssl \
--with-mcrypt --with-mhash \
--with-gmp=/usr/local \
--with-gd=/usr/local/gd \
--with-png-dir=/usr/local \
--with-jpeg-dir=/usr/local \
--with-tiff-dir=/usr/local \
--enable-magic-quotes \
--enable-calendar \
--disable-mbstring \
--with-kerberos=/usr \
--with-freetype-dir=/usr/X11R6 \
--with-xpm-dir=/usr/X11R6 \
--enable-exif \
--enable-ftp \
--enable-bcmath \
--with-pcre-regex=/usr/local/pcre45

make
make install

cheers,

richard
Comment 5 Paul Querna 2004-08-30 07:27:22 UTC
*** Bug 23952 has been marked as a duplicate of this bug. ***
Comment 6 Richard W.M. Jones 2004-09-04 13:06:52 UTC
This bug is really important to fix.  I can't get my mod_caml module
working with Apache 2.0 until it's fixed.  This prevents a significant
userbase from migrating to Apache 2.

My duplicate bug report contains some other information: Bug 23952
Comment 7 Alan Schmitt 2004-09-07 11:00:58 UTC
I just had this problem as well, trying to install a Caml based wiki. If I add
the "LoadModule caml_module modules/mod_caml.so" to my httpd.conf file, apache
just keeps segfaulting.

Unfortunately this is on a server where we need apache2, so this means that
until this fix is integrated, I cannot use mod_caml (and every other mod that
need a recent pcre library, it seems).
Comment 8 Nick Kew 2004-09-16 23:22:38 UTC
Adding PatchAvailable keyword
Comment 9 OpenMacNews 2004-09-18 02:33:26 UTC
just building httpd2.0.51 ...

it seems this bug is STILL not resolved.

there's a demonstrated, working patch (cref: Andreas, below) that fixed the issue in 2.0.50, and 
(seemingly) applies to 2.0.51 as well ... 

is there a particular reason why this known/reported/fixed issue is NOT being integrated?

thx,

richard
Comment 10 Joe Orton 2004-09-18 07:22:05 UTC
There are some subtle issues with this: 

1) with an httpd built to use an external pcre, the include/pcre.h which gets
installed does not match the pcre which is actually used.  It's possible that
some third-party modules will depend on this.

2) Renaming both the reg* and pcre_* functions is a binary compatibility issue.
Comment 11 OpenMacNews 2004-09-30 17:52:29 UTC
hi,

i'm back again!

i'm building httpd 2.0.52 + php 5.0.2 on OSX 10.3.5

the ONLY way i'm able to get httpd & php to 'play nice' together -- avoiding later/'downstream' pcre 
conflicts -- is to make sure that they're built using the SAME version of pcre.

there are two options to that end ...

(a)
        build httpd w/ INTERNAL pcre (v3.9)
        build external pcre3.9
        build php 502 against pcre3.9/external

(b)
        apply Andreas' patch to httpd (*luckily* the patch still works against 2.0.52 ...)
        build external pcre5.0
        build httpd 2.0.52 against pcre5.0/external
        build php 5.0.2 against pcre5.0/external

(a), of course, locks PHP into the use of pcre3.9 ... which is simply not acceptable/tenable for a variety 
of other apps that need >= v4.5

so, (b) is really the only functional option here ...

acknowledging 'possible subleties' from  Joe's reference above, the php issue is neither subtle, nor 
simply possible ... others as well (cref above) are noting problems with, e.g., mod_caml that requires a 
more recent pcre.

what need to be done to get this solution addressed/integrated?  even ASSIGNED would be a good start!

this has been an unaddressed issue problem for QUITE awhile ...

richard
Comment 12 Richard W.M. Jones 2004-09-30 18:00:30 UTC
Agree with the previous comment.

mod_caml cannot use anything except an external, modern PCRE, so somehow
linking it against Apache's ancient version of PCRE is a non-starter,
even if it were possible.

This bug is a showstopper for a variety of important modules, so it
should be fixed (particularly, since there's even a patch to do it).
Comment 13 Richard W.M. Jones 2004-11-10 09:16:24 UTC
Created attachment 13377 [details]
Amended version of patch by Andres Salomon, with typo fixed by Paul Argentoff
Comment 14 Richard W.M. Jones 2004-11-10 09:17:28 UTC
The following notes were sent to me by Paul Argentoff to
accompany the preceeding patch.

  - - - - - -

This is patch by Andres Salomon <dilinger@voxel.net>, with typo fixed by Paul
Argentoff <argentoff@rtelekom.ru>

This patch fixes problems seen by apache modules attempting to run regex
stuff, and getting apache's internal pcre implementation.  It contains
two parts; the first (the autoconf stuff) checks if libpcre is installed
on the host system.  If it is, that pcre is used to build apache2.  If
it's not, apache2 compiles its own pcre, and statically links against
it.  

The second part of this patch forces pcre to be used.  Apache, by
default, calls the posix regex functions (reg{exec,comp,free,error})
internally.  If glibc was sane, this would be fine.  Unfortunately, it's
not.  So, when apache calls these regex functions, it's anyones guess as
to whether it will use glibc's regex functions, or the pcre regex
functions.  The glibc regex functions segfault (within glibc) on my
system.  So, to ensure sanity, we force use of pcre regex functions.

I need to do a bit more testing w/ apache's internal pcre, but for
debian's purposes (pcre in /usr), it works fine.
Comment 15 Owen Gunden 2004-11-23 00:30:23 UTC
Pleeeeeeeeeeeeease incorporate this fix?  I'll bake you cookies!
Comment 16 Andres Salomon 2004-12-06 22:01:08 UTC
Ok, I've created a new patch that should supercede both other patches.  This
patch checks for regex support via libc, and will use that if found.  If the
libc doesn't support POSIX.2 regex functions, it will then check for
an external pcre; if found, it links against that.  If it doesn't find
that, it finally falls back to compiling and linking against its own
internal pcre library.

This patch is also significantly smaller than my other two patches, as it
doesn't require any widespread symbol changes.  I've tested compilation
with and without regex.h and pcre-dev.  Testers whose libc specifically lacks
regex support are requested (all testers welcome, of course :)

I'll follow up w/ fixes as necessary.
Comment 17 André Malo 2004-12-06 22:04:26 UTC
We won't use any POSIX regex stuff. The PCRE is used on purpose...
Comment 18 Andres Salomon 2004-12-06 22:07:12 UTC
Created attachment 13661 [details]
more intelligent patch to link against PCRE

Patch against apache 2.0.52; use libc's regex functions, falling back to
external pcre, and finally building internal pcre only if absolutely necessary.
Comment 19 Andres Salomon 2004-12-06 22:09:44 UTC
(In reply to comment #17)
> We won't use any POSIX regex stuff. The PCRE is used on purpose...

Why not?  What specifically is broken regarding the libc-provided POSIX regex
stuff?  And, why not test for breakage, instead of unconditionally using PCRE
when   it's probably not even necessary?
Comment 20 André Malo 2004-12-06 22:14:37 UTC
Consistency among the various installations and way more power.
Comment 21 Andres Salomon 2004-12-06 22:46:51 UTC
In that case, instead of using the POSIX functions, we should probably just be
using the PCRE functions...
Comment 22 André Malo 2004-12-06 22:55:58 UTC
That's a good point and I personally agree. This should be discussed on dev@
anyway. The best would be with a patch backhand :)
Comment 23 Joe Orton 2004-12-06 23:07:00 UTC
More than that, httpd *cannot* use a real POSIX regex implementation rather than
PCRE: even the default config actually uses some of the Perl extensions to the
regex grammar (the "(?:" stuff IIRC?).
Comment 24 Joe Orton 2005-02-11 16:47:21 UTC
This is all now done on the trunk based on Andres' patches - thanks a lot!

Again, it can't really be backported to 2.0, unfortunately, because it changes
the module interface.
Comment 25 OpenMacNews 2005-02-11 17:04:45 UTC
hi,

> Again, it can't really be backported to 2.0, unfortunately, because it changes the module interface.

so iiuc, then, we've two options at this point:

(1) STABLE 2.0.x release source + Andres' patches (assuming/hoping they continue to apply ...)
(2) UNSTABLE cvs source, w/ functionality included.

i.e., no STABLE solution until 2.1 release ... 

am i correct?

thanks for the update!

richard
Comment 26 apache 2005-02-13 23:32:08 UTC
(In reply to comment #25)

I just tried the most recent patch with Apache 2.0.53 and make failed:

libtool: link: cannot find the library `/Users/me/Desktop/webserver/httpd-2.0.53/srclib/pcre/
libpcre.la'
make[2]: *** [htpasswd] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1
Comment 27 Joe Orton 2005-02-14 15:45:45 UTC
(In reply to comment #25)
> > Again, it can't really be backported to 2.0, unfortunately, because it
changes > > the module interface.
> 
> so iiuc, then, we've two options at this point:

Yes, that's correct.
Comment 28 apache 2005-02-23 16:12:20 UTC
If anyone else is following this, the 2.1.3 alpha is out now, and it solved my problems.
Comment 29 Joe Orton 2005-02-25 17:45:54 UTC
*** Bug 28781 has been marked as a duplicate of this bug. ***
Comment 30 Konstantin Andreev 2005-05-23 20:59:07 UTC
I have applied most recent Andres' patch (id=13661) against most recent current
stable Apache - 2.0.54. Patched sources compile cleanly, but at run-time Apaches
crashes/segfaults immediately after parsing any regex in config file.

Here are some details: in my system this patch selects glibc's implementation of
regex. If there are not regexes in config file, then apache starts successfully.
If there is regex in config file, it compiles without problem. Using gdb,

(gdb) break regcomp
(gdb) r
(gdb) bt
#0  0x403bf852 in regcomp () from /lib/libc.so.6
#1  0x08071714 in ap_pregcomp (p=0x809b0a8, pattern=0x80d3bd0 "^\\.ht",
cflags=135085008) at util.c:268
#2  0x0807ca4e in filesection (cmd=0xbffff990, mconfig=0x80d3bd0, arg=0x80d3bc1
"") at core.c:1837
#3  0x0806b5f1 in invoke_cmd (cmd=0x808e2e8, parms=0xbffff990,
mconfig=0x80b59f0, args=0x80d2b90 "~ \"^\\.ht\">")
    at config.c:797
#4  0x0806bf06 in ap_walk_config (current=0x80d2b70, parms=0xbffff990,
section_vector=0x80b58e0) at config.c:1060
#5  0x0806cd2f in ap_process_config_tree (s=0x80d3bd8, conftree=0x80d3bd0,
p=0x809b0a8, ptemp=0x80d3bd0)
    at config.c:1643
#6  0x0806fa6b in main (argc=5, argv=0xbffffab4) at main.c:539

But, shortly, segfaults occurs:
(gdb) cont
Continuing.

Program received signal SIGSEGV, Segmentation fault.
trie_node_alloc (p=0x809b0a8, parent=0x809b518, c=95 '_') at util_filter.c:113
113                 if (c == parent->children[i].c) {
(gdb) bt
#0  trie_node_alloc (p=0x809b0a8, parent=0x809b518, c=95 '_') at util_filter.c:113
#1  0x080766a3 in register_filter (name=0x809b518 "core_in", filter_func=
      {out_func = 0x807ed90 <core_input_filter>, in_func = 0x807ed90
<core_input_filter>}, filter_init=0, ftype=0, 
    reg_filter_set=0x809b518) at util_filter.c:217
#2  0x08080611 in register_hooks (p=0x809b0a8) at core.c:4509
#3  0x0806fb4f in main (argc=5, argv=0xbffffab4) at main.c:578
Comment 31 Andres Salomon 2005-05-23 21:49:29 UTC
This should be fixed in apache-2.1; Joe Orton incorported my patch into it, plus
added some additional fixes.  If you're going to use 2.0, don't use my patches
(I'm not keeping them up-to-date); use Debian's.  Adam Conrad backported Joe's
changes for 2.0, and they're being kept up-to-date.

ftp://ftp.debian.org/debian/pool/main/a/apache2/

apt-get source apache2, or grab the patch out of the .diff.gz.