Sending a request via a mod_proxy reverse proxy to an https backend results in a segfault. This is a similar issue to bug 24030 in that the segfault also occurs in ssl_callback_proxy_cert calling the modssl_set_cert_info() macro: #define modssl_set_cert_info(info, cert, pkey) \ *cert = info->x509; \ X509_reference_inc(*cert); \ *pkey = info->x_pkey->dec_pkey; \ EVP_PKEY_reference_inc(*pkey) except that instead of info->x_pkey being NULL, it is info->x_pkey->dec_pkey (the private key data) that is NULL. (gdb) p *info $1 = {x509 = 0xcb2140, crl = 0x0, x_pkey = 0xcaf490, enc_cipher = {cipher = 0x0, iv = "?\"?\000\000\000\000\000?\t?\000\000\000\000"}, enc_len = 0, enc_data = 0x0, references = 1} (gdb) set print pretty on (gdb) p info->x_pkey->dec_pkey $5 = (EVP_PKEY *) 0x0 (gdb) c Continuing. Program received signal SIGSEGV, Segmentation fault. 0x0000002a96f10e82 in CRYPTO_add_lock (pointer=0x8, amount=1, type=5, file=0x2a96fbdd7f "ssl_engine_kernel.c", line=1698) at cryptlib.c:630 ^Z^Z/app/home/calrn13t/dl/openssl-1.0.0e/crypto/cryptlib.c:630:18583:beg:0x2a96f10e82 (gdb) bt #0 0x0000002a96f10e82 in CRYPTO_add_lock (pointer=0x8, amount=1, type=5, file=0x2a96fbdd7f "ssl_engine_kernel.c", line=1698) at cryptlib.c:630 #1 0x0000002a96edd0a3 in ssl_callback_proxy_cert (ssl=0xcbc710, x509=0x7fbfffea28, pkey=0x7fbfffea30) at ssl_engine_kernel.c:1698 #2 0x0000002a96f09deb in ssl_do_client_cert_cb (s=0xcbc710, px509=0x7fbfffea28, ppkey=0x7fbfffea30) at s3_clnt.c:3048 #3 0x0000002a96f09e89 in ssl3_send_client_certificate (s=0xcbc710) at s3_clnt.c:2812 #4 0x0000002a96f0a378 in ssl3_connect (s=0xcbc710) at s3_clnt.c:373 ... I initially avoided the segfault by modifying the fix for bug 24030 to check x_pkey->dec_pkey as well: --- a/httpd-2.2.21/modules/ssl/ssl_engine_init.c +++ b/httpd-2.2.21/modules/ssl/ssl_engine_init.c @@ -1006,7 +1006,7 @@ static void ssl_init_proxy_certs(server_rec *s, for (n = 0; n < ncerts; n++) { X509_INFO *inf = sk_X509_INFO_value(sk, n); - if (!inf->x509 || !inf->x_pkey) { + if (!inf->x509 || !inf->x_pkey || !inf->x_pkey->dec_pkey) { sk_X509_INFO_free(sk); ap_log_error(APLOG_MARK, APLOG_STARTUP, 0, s, "incomplete client cert configured for SSL proxy " which resulted in a: incomplete client cert configured for SSL proxy (missing or encrypted private key?) error on startup, which was an improvement. Further investigation using gdb and testing showed that having the private key before the certificate in SSLProxyMachineCertificateFile was triggering the segfault. Changing this file to have the certificate first resolved the issue. I'd suggest applying the above patch to avoid the segfault and at least updating the SSLProxyMachineCertificateFile documentation to say that the certificate should come before the private key. Note that this issue does not occur in Apache/2.2.11 with OpenSSL/0.9.8k but does also occur in Apache/2.2.17 with OpenSSL/1.0.0c.
Seeing this on EL6 with mod_ssl-2.2.15-15.sl6.1.i686
Thanks for the patch, Keith. I've committed this with an added check that the keypair actually matches. r1374214
(In reply to comment #0) > Further investigation using gdb and testing showed that having the private > key before the certificate in SSLProxyMachineCertificateFile was triggering > the segfault. Changing this file to have the certificate first resolved the > issue. > > I'd suggest applying the above patch to avoid the segfault and at least > updating the SSLProxyMachineCertificateFile documentation to say that the > certificate should come before the private key. > > Note that this issue does not occur in Apache/2.2.11 with OpenSSL/0.9.8k but > does also occur in Apache/2.2.17 with OpenSSL/1.0.0c. This observation is correct. It's actually caused by a regression in OpenSSL 1.0.0 and later, as reported here, in the meantime: http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3028 With OpenSSL up to 0.9.8, the order doesn't matter. In this case, however, the check for encrypted private keys is insufficient, as found when dealing with bug 54698. An additional fix has been committed to trunk in r1467593. Backport proposals for 2.2.x and 2.4.x submitted with r1467594.
Backported to v2.4.
Commit for 2.4.x: r1476685. To appear in 2.4.5.
(In reply to Kaspar Brand from comment #3) > It's actually caused by a regression in OpenSSL > 1.0.0 and later, as reported here, in the meantime: > > http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3028 Will be fixed with the next OpenSSL releases, i.e. 1.0.0l, 1.0.1f and 1.0.2 (committed to the repository in early August).