Bug 57360 - Fail gracefully on certificate/key mismatch
Summary: Fail gracefully on certificate/key mismatch
Status: NEW
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_ssl (show other bugs)
Version: 2.4.10
Hardware: PC Linux
: P2 normal with 3 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-12-17 05:20 UTC by Michael Orlitzky
Modified: 2023-01-06 09:06 UTC (History)
5 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Orlitzky 2014-12-17 05:20:26 UTC
From the mod_ssl docs:

> SSLCertificateChainFile is deprecated
>
> SSLCertificateChainFile became obsolete with version 2.4.8, when 
> SSLCertificateFile was extended to also load intermediate CA certificates from 
> the server certificate file.

Now that this is the case, there's a very easy mistake one can make that will crash the server. When combining the CA chain and site certficate files, if one does,

  $ cat chain.crt site.crt > combined.crt

instead of,

  $ cat site.crt chain.crt > combined.crt

then the server will crash on the next graceful reload. It will also refuse to start; the only thing logged is a cryptic "AH00016: Configuration Failed" which is misleading at best.

I don't know whether it's a good idea to proceed with one dead vhost -- the site in question obviously won't work with a mismatched key/cert -- but if not a better error message would be nice. I spent rather a long time searching for other problems while all of our sites were down because it never occurred to me that a key/cert mismatch could crash the whole server.
Comment 1 Sophie Herold 2015-11-25 10:38:16 UTC
I ran into the same problem on a production system. I did not expect apache2 to crash during a reload operation. Such behavior does not really help in expanding the adoption of encryption. With imminent automation through "Let's encrypt" this can even break systems without an admin directly being involved.

However, I do get a syslog entry

> [ssl:emerg] [pid 10249] AH02565: Certificate and private key 127.0.0.1:443:0 from /etc/apache2/ssl/test.crt and /etc/apache2/ssl/test.key do not match

with configured "ErrorLog syslog:user". But I think that "emergency" is _not_ the correct severity level.

Possible Solutions (without knowing if it is technically feasible):

1. Catch the Openssl signal and disable vhost and log
2. Catch the Openssl signal and recover with emergency key and certificate (config values) and log, disable vhost if emergency key/cert are not available or cause error with openssl
3. Involve Openssl in the configtest and let reload fail if there is any kind of cert problem
Comment 2 Sophie Herold 2015-11-25 16:00:33 UTC
I only proposed 1. and 2. since I thought 3. might be complicated. However, all the tools required to do the necessary checks at configtest are in place.

Currently, the certificate/key checks are done in ssl_init_server_certs which is called as post config hook (ap_hook_post_config).

Probably, moving one part of this code to a check config hook (ap_hook_check_config) would solve the problem. However, the support of encrypted private keys makes this task non-trivial (at least for me). I would have to investigation further to see if this a real problem.
Comment 3 James Boyle 2017-03-30 16:10:10 UTC
I've experienced this issue as well - I just want to share what I believe to be a common expectation among system administrators / DevOps folks out there.  When a daemon fails to load, we expect the reason for the failure to be clearly logged, ideally to syslog.  I agreed that the "Configuration Failed" (aka Apache initialization) and Apache returned "1" when it exited, but I hope everyone will admit the following two statements don't really give the administrator a good place to look for things to fix:

"AH00016: Configuration Failed" ... and ...
"Syntax OK" (from httpd -t) - which implies the configuration is OK

I wish it had said something like:
"AH00016: Configuration Failed: Security certificate error: /path/to/file"
or
"AH00016: Configuration Failed: VHost example.com: cannot initialize SSL/TLS"

Thankfully I know about strace.

Any solution that puts a message that points the administrator to the source of the problem is a win in my book.

Thank you!
Comment 4 Andreas Schnederle-Wagner 2019-09-20 10:07:51 UTC
This really odd behaviour just brought down one of our CMS Servers resulting in ~1.000 Websites being down for ~30 Minutes ...

We rely on automated "configtest" before doing "reload" - it should NEVER happen that Apache refuses to start if neither configtest nor reload show any sign of Problem ...

IMHO this should be handled with higher Priority than P2/normal ...
Comment 5 Andrew G. Watters 2023-01-05 15:50:14 UTC
This is still a bug as of 2022.  I was panicking because I manage several websites, all of which are under one certificate and were down for 45 minutes while I troubleshooted.  It would be really helpful if regular HTTP could start without HTTPS in the event of a mismatch.

The cryptic startup error messages eventually led me to the ssl_error_log, which reported:

AH02565: Certificate and private key fe80::3eec:efff:fefb:56d6:443:0 from /mnt/raid5/etc/ssl/andrewwatters_com.crt and /mnt/raid5/etc/ssl/andrewwatters_com.key do not match

This apparently happens when the Certificate Signing Request uses the wrong key.  To correct the CSR and use it with your existing key, do this:

openssl req -new -key yourkey.key -out req.csr

You'll have to fill out the required CSR fields, and you'll get a corrected CSR to use with your SSL provider.  Hopefully they can promptly reissue the certificate or else you have a few days left on your prior certificate.
Comment 6 Joe Orton 2023-01-06 09:06:37 UTC
FWIW I agree it's a bug this isn't caught by "httpd -t", but it would definitely not be acceptable to silently "disable" some vhosts because the certs were badly configured.

Looks like it might require some significant surgery to fix, and there are various different ways certs can be configured, so doesn't look like a trivial patch at all.