This happens since I upgraded Apache from 2.4.7 to 2.4.9. The error i get looks like this: [Sun Mar 23 23:33:13.231712 2014] [mpm_event:notice] [pid 21463:tid 140220024149824] AH00493: SIGUSR1 received. Doing graceful restart [Sun Mar 23 23:33:13.290939 2014] [auth_digest:notice] [pid 21463:tid 140220024149824] AH01757: generating secret for digest authentication ... [Sun Mar 23 23:33:13.292180 2014] [ssl:error] [pid 21463:tid 140220024149824] AH02217: ssl_stapling_init_cert: Can't retrieve issuer certificate! [Sun Mar 23 23:33:13.292187 2014] [ssl:error] [pid 21463:tid 140220024149824] AH02567: Unable to configure certificate rtmp.btbn.de:443:0 for stapling [Sun Mar 23 23:33:13.292811 2014] [ssl:emerg] [pid 21463:tid 140220024149824] (2)No such file or directory: AH02574: Init: Can't open server private key file 8\xd1\x9a [Sun Mar 23 23:33:13.292833 2014] [ssl:emerg] [pid 21463:tid 140220024149824] AH02312: Fatal error initialising mod_ssl, exiting. [Sun Mar 23 23:33:13.292836 2014] [ssl:emerg] [pid 21463:tid 140220024149824] AH02564: Failed to configure encrypted (?) private key sync.btbn.de:443:1, check 8\xd1\x9a [Sun Mar 23 23:33:13.292839 2014] [:emerg] [pid 21463:tid 140220024149824] AH00020: Configuration Failed, exiting [Sun Mar 23 23:33:41.017925 2014] [ssl:error] [pid 21895:tid 140712203999040] AH02217: ssl_stapling_init_cert: Can't retrieve issuer certificate! [Sun Mar 23 23:33:41.017995 2014] [ssl:error] [pid 21895:tid 140712203999040] AH02567: Unable to configure certificate rtmp.btbn.de:443:0 for stapling [Sun Mar 23 23:33:41.019157 2014] [ssl:emerg] [pid 21895:tid 140712203999040] (2)No such file or directory: AH02574: Init: Can't open server private key file 8\xc1\xf2\x01 [Sun Mar 23 23:33:41.019176 2014] [ssl:emerg] [pid 21895:tid 140712203999040] AH02312: Fatal error initialising mod_ssl, exiting. [Sun Mar 23 23:33:41.019181 2014] [ssl:emerg] [pid 21895:tid 140712203999040] AH02564: Failed to configure encrypted (?) private key sync.btbn.de:443:1, check 8\xc1\xf2\x01 AH00016: Configuration Failed I am able to workaround this by putting SSLCertificateKeyFile and/or SSLCertificateFile in each single ssl enabled vhost. It doesn't matter which one i add(key or cert), just stating at least one of them makes the error disappear. I have a globaly configured wildcard cert which matches most of my vhosts, and only override it in those where it doesn't apply, that's why most of my vhosts didn't have an individual cert configured before. It seems to be possible to reproduce it by just configuring a few vhosts which use ssl without specifiying a cert/key inside of them, while having both globaly configured. I am currently not able to test that, because i don't have a test machine available.
(In reply to Timo R. from comment #0) > It seems to be possible to reproduce it by just configuring a few vhosts > which use ssl without specifiying a cert/key inside of them, while having > both globaly configured. I am currently not able to test that, because i > don't have a test machine available. I'm not able to reproduce the problem with such a config, specifically, but I realize that a side effect of r1573360 is that the behavior when merging global and per-vhost SSLCertificateFile/SSLCertificateKeyFile directives is different now (cf. https://svn.apache.org/viewvc/httpd/httpd/branches/2.4.x/modules/ssl/ssl_engine_config.c?r1=1573360&r2=1573359&pathrev=1573360 for the specific changes). Could you provide the skeleton of your configuration, which only shows the global and per-vhost SSLCertificateFile and SSLCertificateKeyFile directives? Or as a first step, the output of "/httpd -t -D DUMP_VHOSTS -D DUMP_CERTS" perhaps?
The output with the workaround in place is: http://bpaste.net/show/194089/ In a state where it fails it looks like this: http://bpaste.net/show/194091/ The condensed config looks like this: global: SSLSessionCache shmcb:/run/ssl_scache(512000) SSLSessionCacheTimeout 300 SSLUseStapling on SSLStaplingCache shmcb:/run/ssl_stapling_cache(512000) SSLHonorCipherOrder On SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH Mutex file:/run/apache_ssl_mutex ssl-cache SSLCertificateFile /etc/ssl/private/server.pem SSLCertificateKeyFile /etc/ssl/private/server.key vhosts: <VirtualHost *:443> ServerName btbn.de Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName zkb.mtl-eve.org Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName www.mtl-eve.org Include /etc/apache2/myvhosts.d/ssl-startcom.include SSLCertificateFile /etc/ssl/private/mtl-eve.pem </VirtualHost> <VirtualHost *:443> ServerName myadmin.oromit.de Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName www.project-phaethon.de Include /etc/apache2/myvhosts.d/ssl-startcom.include SSLCertificateFile /etc/ssl/private/project-phaethon.pem </VirtualHost> <VirtualHost *:443> ServerName ci.btbn.de Include /etc/apache2/myvhosts.d/ssl-startcom.include SSLCertificateFile /etc/ssl/private/ci.pem </VirtualHost> <VirtualHost *:443> ServerName git.btbn.de Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName bts.btbn.de Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName sync.btbn.de Include /etc/apache2/myvhosts.d/ssl-startcom.include SSLCertificateFile /etc/ssl/private/sync.pem </VirtualHost> <VirtualHost *:443> ServerName rtmp.btbn.de Include /etc/apache2/myvhosts.d/ssl.include </VirtualHost> <VirtualHost *:443> ServerName obsproject.org Include /etc/apache2/myvhosts.d/ssl-startcom.include SSLCertificateFile /etc/ssl/private/obsproject.pem </VirtualHost> ssl-common.include: SSLEngine on #SSLCertificateKeyFile /etc/ssl/private/server.key # Enabling this line fixes the problem ssl.include: Include ssl-common.include #SSLCertificateFile /etc/ssl/private/server.pem # Enabling this line also fixes it ssl-startcom.include: Include ssl-common.include SSLCACertificateFile /etc/apache2/startcom/ca.pem
Created attachment 31458 [details] Only read "active" values from the mctx->pks->key_files array (In reply to Timo R. from comment #2) Thank you for the detailed information. > The output with the workaround in place is: > > http://bpaste.net/show/194089/ I assume that you disabled the two global-level SSLCertificateFile and SSLCertificateKeyFile directives for this case, is that correct? (The "Server certificates" list includes 11 entries, i.e. one per <VirtualHost *:443> block.) > In a state where it fails it looks like this: > > http://bpaste.net/show/194091/ Here we find 27 entries in the server certificates list (and still 11 <VirtualHost *:443> blocks), so it's obvious that merging the global and the per-vhost settings has somewhat surprising effects. The three-layer Include structure in your config makes it relatively hard to follow what SSLCertificate/SSLCertificateKeyFile directives are active for which VirtualHost (and what config exactly was used for the output at http://bpaste.net/show/194091/), but from looking at the code in ssl_engine_init.c and the errors shown in the description above, the problem seems to be that we try to read a bogus value for the SSLCertificateKeyFile directive. Could you try to apply the attached patch to 2.4.9? This should make the garbled log messages go away - but it might not yet fix the underlying issue, which probably needs further work (in particular when looking at the order of merging the global and per-vhost settings).
Created attachment 31459 [details] Only read "active" values from the mctx->pks->key_files array, v2 (In reply to Kaspar Brand from comment #3) > Could you try to apply the attached patch to 2.4.9? Hmpf, this was off by one, sorry. Please try this version instead.
(In reply to Kaspar Brand from comment #4) > Created attachment 31459 [details] > Only read "active" values from the mctx->pks->key_files array, v2 Committed to trunk with r1585918 and proposed for backport to 2.4.x in r1585922. Timo, if you are able to try the patch with your setup and 2.4.9, a report on your tests is still appreciated.
Oh, i completely forgot about this one. Will test it later today.
With the patch applied, it still does not start up, but the garbled part is gone. Instead i get: [Wed Apr 09 13:46:09.931766 2014] [ssl:error] [pid 29696:tid 139644707223360] AH02217: ssl_stapling_init_cert: Can't retrieve issuer certificate! [Wed Apr 09 13:46:09.931809 2014] [ssl:error] [pid 29696:tid 139644707223360] AH02567: Unable to configure certificate rtmp.btbn.de:443:0 for stapling [Wed Apr 09 13:46:09.932141 2014] [ssl:error] [pid 29696:tid 139644707223360] AH02217: ssl_stapling_init_cert: Can't retrieve issuer certificate! [Wed Apr 09 13:46:09.932146 2014] [ssl:error] [pid 29696:tid 139644707223360] AH02567: Unable to configure certificate sync.btbn.de:443:0 for stapling [Wed Apr 09 13:46:09.949913 2014] [ssl:error] [pid 29696:tid 139644707223360] AH02579: Init: Private key not found [Wed Apr 09 13:46:09.949940 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag [Wed Apr 09 13:46:09.949950 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error [Wed Apr 09 13:46:09.949958 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag [Wed Apr 09 13:46:09.949967 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=RSA) [Wed Apr 09 13:46:09.949975 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib [Wed Apr 09 13:46:09.949982 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag [Wed Apr 09 13:46:09.949990 2014] [ssl:error] [pid 29696:tid 139644707223360] SSL Library Error: error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO) [Wed Apr 09 13:46:09.949994 2014] [ssl:emerg] [pid 29696:tid 139644707223360] AH02312: Fatal error initialising mod_ssl, exiting. [Wed Apr 09 13:46:09.949998 2014] [ssl:emerg] [pid 29696:tid 139644707223360] AH02564: Failed to configure encrypted (?) private key sync.btbn.de:443:1, check /etc/ssl/private/server.pem AH00016: Configuration Failed The stapling errors don't seem to be part of the issue. The server.pem file does not contain a private key. Plase note that my server configuration has changed in the meantime(due to Heartbleed and StartCom beeing bad). All vhosts use the same cert and key now. If no vhost has a Cert and Key defined, so only the global directives exist, it works(Starts up, erroring only about the stapling). In order to replicate the issue i put an additional SSLCertificateFile into some of the vhosts.
Created attachment 31531 [details] Restore previous SSLCertificate[Key]File directive merging behavior for 2.4.x Thank you for testing, and sorry for being late with my reply. The stapling errors are unrelated, indeed. The root cause of this issue is what I already mentioned in comment 1: with r1573360 (i.e. httpd 2.4.8/2.4.9), merging of global and per-vhost SSLCertificate[Key]File directives inadvertently changed in a non-backwards compatible way. With the attached patch (to be applied in addition to the first one), I tried to restore the previous behavior as far as possible. The most important difference to 2.4.9 is that an SSLCertificate[Key]File directive in a VirtualHost block now again overrides a global setting (in 2.4.9, the global setting is appended to the list, and then mistakenly replaces the per-vhost setting, see also bug 56353). Testing of your setup (possibly with the one you originally reported) with this patch applied would be very much appreciated.
(In reply to Kaspar Brand from comment #5) > proposed for backport to 2.4.x in r1585922. Committed to 2.4.x with r1588246. To appear in 2.4.10. This fixes the specific issue reported in this bug, but doesn't restore backwards compatibility with versions before 2.4.8 yet (merging of global and vhost-level directives). Based on feedback from testing the patch in comment 8, I'm intending to propose an additional backport for 2.4.x.
Comment on attachment 31531 [details] Restore previous SSLCertificate[Key]File directive merging behavior for 2.4.x Obsolete in this form, as it is included in the https://people.apache.org/~kbrand/mod_ssl-2.4.x-pphrase-certkeyfile-compat.diff backport proposal
(In reply to Kaspar Brand from comment #9) > I'm intending to propose an additional backport for 2.4.x. I've done so now with r1590359. Examining again the configuration from comment 2, I can also provide an explanation for the troubles you experienced when using your existing (pre-2.4.9) setup: as already mentioned in comment 8, a vhost-level SSLCertificate[Key]File directive is overriding a globally configured one in releases up to 2.4.7, but in 2.4.8/2.4.9, it is simply added to the list. In your case, you had five VirtualHost blocks with an individual SSLCertificateFile directive, but no corresponding SSLCertificateKeyFile, and mod_ssl was trying to read the file name from an uninitialized key_files array value. The immediate fix for this has already been committed (see comment 9), but the additional patch in https://people.apache.org/~kbrand/mod_ssl-2.4.x-pphrase-certkeyfile-compat.diff will further restore backwards compatibility for existing configurations.
(In reply to Kaspar Brand from comment #11) > (In reply to Kaspar Brand from comment #9) > > I'm intending to propose an additional backport for 2.4.x. > > I've done so now with r1590359. This has now been committed to the 2.4.x branch with r1593003 - will appear in 2.4.10.
Fixed and released in 2.4.10