Bug 43334

Summary: Garbage characters in Server header and version string when using mod_ssl statically
Product: Apache httpd-2 Reporter: Steven Chamberlain <steven>
Component: mod_sslAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: critical CC: c.hargr, Craig, cuicui.oizo, durket, rl, rudy.amid, srattai, wbreyha
Priority: P1 Keywords: PatchAvailable
Version: 2.2.6   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
URL: http://svn.apache.org/viewvc?view=rev&revision=574884

Description Steven Chamberlain 2007-09-09 09:43:12 UTC
Since upgrading apache to 2.2.6, I now see 'garbage' displayed in the Server
header of the HTTP response, which I believe is caused by mod_ssl:
  Server: Apache/2.2.6 (Unix) mod_ssl/2.2.6 (8$ DAV/2

The error log contains nothing other than this notice, also showing the garbage
characters:
  [notice] Apache/2.2.6 (Unix) mod_ssl/2.2.6 (8$\b DAV/2 configured -- resuming
normal operations

The garbage characters do not change during successive HTTP requests.  The
string *sometimes* varies when httpd is restarted via 'apachectl restart'.  A
space (0x20) always appears at either side of the garbage characters.  Sometimes
after restarting, no garbage characters appear but there are still two spaces
between mod_ssl/2.2.6 and DAV/2 (presumably the garbage output began with 0x00
on those occasions).

The error is present when no shared modules are being loaded.  I had compiled
httpd from source and configured as follows:
  ./configure --prefix= --localstatedir=/var --sysconfdir=/etc/apache2
--enable-layout=Debian --enable-so --with-program-name=apache2
--with-suexec-caller=www-data --with-suexec-bin=/usr/lib/apache2/suexec2
--with-suexec-docroot=/var/www --with-suexec-userdir=public_html
--with-suexec-logfile=/var/log/apache2/suexec.log --with-ldap=yes
--with-ldap-include=/usr/include --with-ldap-lib=/usr/lib --with-z
--enable-deflate --enable-headers --with-mpm=worker --enable-expires
--enable-ssl --enable-dav --with-apr=/usr

After compiling without --enable-ssl, the version string appeared normal:
  [notice] Apache/2.2.6 (Unix) DAV/2 configured -- resuming normal operations

To double-check, I recompiled with --enable-ssl and restarted, and the problem
reappeared:
  [notice] Apache/2.2.6 (Unix) mod_ssl/2.2.6 \xde\x10\x10\b\xc6\x10\x10\b DAV/2
configured -- resuming normal operations

An interesting side-effect is that the garbage characters can trigger this error
in Visual Studio or .NET HTTP clients:
  "The server committed a protocol violation. Section=ResponseHeader Detail=CR
must be followed by LF"
The error message partly erroneous, since indeed all CR's were followed by LF's,
and the garbage characters at the time did not include either the CR or LF
control characters.  However, some of the garbage characters were probably
invalid for an HTTP response header, hence the 'protocol violation').

Possibly related to #40146 ?  Though I'm not sure what the "Current
configuration:" message is that the author referred to.

ps. the Version field in the bug tracker does not include 2.2.6, so I had to
select 2.2-HEAD.

Thanks!
Comment 1 Steven Chamberlain 2007-09-09 09:52:15 UTC
I now realise that the the string 'OpenSSL/x.x.x' should be appearing in place
of the garbage characters.  I am using the latest available precompiled package
from Debian 'etch' (stable), named libssl0.9.8.

  # ldd `which apache2`
        ...
        libssl.so.0.9.8 => /usr/lib/i686/cmov/libssl.so.0.9.8 (0xb7f48000)
        ...
  # dpkg -S /usr/lib/i686/cmov/libssl.so.0.9.8
  libssl0.9.8: /usr/lib/i686/cmov/libssl.so.0.9.8
  # dpkg -s libssl0.9.8
  Package: libssl0.9.8
  ...
  Version: 0.9.8c-4

  # openssl version
  OpenSSL 0.9.8c 05 Sep 2006
Comment 2 Ruediger Pluem 2007-09-09 12:11:14 UTC
Could you please try if the following makes the error go away?:

Index: ssl_engine_vars.c
===================================================================
--- ssl_engine_vars.c   (Revision 573732)
+++ ssl_engine_vars.c   (Arbeitskopie)
@@ -640,7 +640,7 @@
     static char *library = NULL;
     char *result;

-    if (!library) {
+    {
         char *cp, *cp2;
         library = apr_pstrdup(p, SSL_LIBRARY_DYNTEXT);
         if ((cp = strchr(library, ' ')) != NULL) {

This patch is no solution, but should help finding the root cause. I for myself
cannot reproduce this currently.
Comment 3 Steven Chamberlain 2007-09-09 12:30:35 UTC
Thanks!  That actually did work:
  [notice] Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.8c DAV/2 configured --
resuming normal operations

I can see that the patch is just a workaround and that something bad is
happening... let me know if there's anything else I can do to find the cause of
this.
Comment 4 Ruediger Pluem 2007-09-09 13:06:07 UTC
(In reply to comment #2)

> Index: ssl_engine_vars.c
> ===================================================================
> --- ssl_engine_vars.c   (Revision 573732)
> +++ ssl_engine_vars.c   (Arbeitskopie)
> @@ -640,7 +640,7 @@
>      static char *library = NULL;
>      char *result;
> 
> -    if (!library) {
> +    {
>          char *cp, *cp2;
>          library = apr_pstrdup(p, SSL_LIBRARY_DYNTEXT);

I assume that somehow we use the wrong pool when we call
ssl_var_lookup_ssl_version for the first time. But this needs to be examined
further. Thanks for your quick feedback.

Comment 5 Kerem Erkan 2007-09-10 23:41:15 UTC
Hi,

We are using both Debian 3.1 and Debian 4.0 servers, and this problem only
happens on Debian 4.0 servers with OpenSSL 0.9.8 installed.

This is what httpd 2.2.6 on Debian 3.1 reports when starting:

Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.7e configured -- resuming normal
operations

And this is what httpd 2.2.6 on Debian 4.0 reports:

Apache/2.2.6 (Unix) mod_ssl/2.2.6 \x14\xf1\x0f\b configured -- resuming normal
operations

Both binaries are configured with the same options and mod_sll is included
statically.
Comment 6 William A. Rowe Jr. 2007-09-12 03:49:16 UTC
Try the committed patch (see URL) against 2.2.6 - this already looks
good against trunk.
Comment 7 Zvi Har'El 2007-09-12 04:18:58 UTC
tested patch against 2.2.6 in RHEL 4 and Solaris 8/9. Works fine. 
Comment 8 Steven Chamberlain 2007-09-12 05:49:27 UTC
Patch applies OK against 2.2.6, and fixes the bug on Debian 4.0 (etch).  Thanks!
Comment 9 William A. Rowe Jr. 2007-09-12 06:54:52 UTC
Things don't go so well applying the patch from trunk onto 2.0 - an adjusted patch
can be grabbed from

http://people.apache.org/~wrowe/ssl_version_pool_fix-2.0.patch

I'll mark fixed when applied to the bugged branches 2.0 and 2.2
Comment 10 William A. Rowe Jr. 2007-09-13 06:56:57 UTC
This has been applied to 2.0 and 2.2 working branches.  Fix upcoming in the next
releases.
Comment 11 Marcus Neukert 2007-09-26 09:18:41 UTC
This Bug is critical, because the garbage in the Server-Header can contain
line-feeds or other stuff. In this case, the server-response is broken. For
example, if the garbage contains line-feeds, all following headers are part of
the response-body.
Comment 12 William A. Rowe Jr. 2007-10-04 22:27:43 UTC
Patch applied to both branches.
Comment 13 Takashi Sato 2007-10-24 16:02:46 UTC
*** Bug 43695 has been marked as a duplicate of this bug. ***
Comment 14 Takashi Sato 2007-10-26 09:23:37 UTC
*** Bug 43708 has been marked as a duplicate of this bug. ***
Comment 15 Joe Orton 2007-11-16 02:40:05 UTC
*** Bug 43865 has been marked as a duplicate of this bug. ***
Comment 16 Ruediger Pluem 2007-11-20 12:19:42 UTC
*** Bug 43918 has been marked as a duplicate of this bug. ***
Comment 17 Ruediger Pluem 2008-01-02 05:46:16 UTC
*** Bug 44161 has been marked as a duplicate of this bug. ***
Comment 18 Takashi Sato 2008-01-17 05:13:37 UTC
*** Bug 44258 has been marked as a duplicate of this bug. ***
Comment 19 Craig 2008-01-17 06:27:50 UTC
As this is a critial bug (can break the correct functionality of apache; also
may disclose paths, rewrite rules and other things from the memory), there
should be a new release of apache, 2.0.62!
I'm a bit disappointed to see 2.0.61 happily announced as "stable" version! :(
Comment 20 Takashi Sato 2008-01-17 08:47:18 UTC
*** Bug 44260 has been marked as a duplicate of this bug. ***