A certificate and private key created by Java's keytool and extracted using various tools for use with the SSLProxyMachineCertificateFile directive causes Apache2's child process (on Windows) to crash (and then reload). Here are the error messages printed to the error.log: [Fri Oct 22 15:09:16 2004] [notice] Parent: child process exited with status 3221225477 -- Restarting. [Fri Oct 22 15:09:17 2004] [notice] Parent: Created child process 1488 [Fri Oct 22 15:09:19 2004] [notice] Child 1488: Child process is running [Fri Oct 22 15:09:19 2004] [notice] Child 1488: Acquired the start mutex. [Fri Oct 22 15:09:19 2004] [notice] Child 1488: Starting 250 worker threads. I will post instructions on how to reproduce this error, sample files, and a work-around next.
To reproduce the error, a certificate and key must be created using Java's keytool, then the cert and key extracted using various means. I used information from this web page to help me in the extraction process: http://mark.foster.cc/kb/openssl-keytool.html Also note that I'm using this pre-compiled distribution to test: http://hunter.campbus.com/ http://hunter.campbus.com/Apache_2.0.52-Openssl_0.9.7d-Win32.zip Finally, I am hitting a Tomcat 5.0.28 server on the back end that is configured to require client authentication (referencing the JKS keystore created in step 1). 1. Create a self-signed certificate and key in JKS format using Java's keytool. I used j2sdk1.4.2_06 for Win32: D:\> D:\j2sdk1.4.2_06\bin\keytool -genkey -alias crashapache2 -keyalg RSA -keystore 31856.jks Password: pass12 First and Last Name: Crash Apache2 Name of Organizational Unit: Bugzilla Name of Organization: ASF Name of City: Forest Hill Name of State: Maryland Two-letter ISO country: US Is [...] correct? yes Enter password for <crashapache2>: [hit Enter] 2. Use keytool to extract the certificate in DER format: D:\> D:\j2sdk1.4.2_06\bin\keytool -export -alias crashapache2 -keystore 31856.jks -file crashapache2-der.crt Enter keystore password: pass12 3. Use OpenSSL to convert the DER-encoded certificate to PEM-encoded: D:\> D:\Apache2\bin\openssl x509 -out crashapache2.crt -outform pem -text -in crashapache2-der.crt -inform der 4. Use the ExportPriv.java class (http://mark.foster.cc/pub/java/ExportPriv.java) to extract the private key. D:\> D:\j2sdk1.4.2_06\bin\javac ExportPriv.java D:\> D:\j2sdk1.4.2_06\bin\java ExportPriv 31856.jks crashapache2 pass12 > crashapache2.key 5. Combine the crashapache2.crt and crashapache2.key files into a single file, crashapache2.crtkey, and remove the extraneous text so that the file looks like this: -----BEGIN CERTIFICATE----- MII... -----END CERTIFICATE----- -----BEGIN PRIVATE KEY----- MII... -----END PRIVATE KEY----- IMPORTANT NOTE: The file must contain UNIX-style line endings! If DOS-style line endings are used, the file will not be parsed properly. (I used Cygwin and vim to accomplish this.) 6. Reference the crashapache2.crtkey file in the SSLProxyMachineCertificateFile directive, restart Apache, then try to access a protected URL. The error messages should be printed to error.log (assuming there is an error log defined outside of all virtual hosts). IMPORTANT NOTE: The remote server MUST be set up properly to use the same certificate and private key, or else you will NOT see this error message. Instead, this message will be printed to the virtual-host-specific error log: [Fri Oct 22 16:08:42 2004] [error] (20014)Error string not specified yet: proxy: request failed to W.X.Y.Z:8443 (fully.qualified.host.name)
In step 6, when I said that the remote server must be, "set up properly to use the same certificate and private key", I meant that the remote server must be configured to expect the same certificate and private key to be used for client authentication. The remote server needs a different certificate and private key to communicate via https. I will note this when uploading the Tomcat server.xml config file later.
The workaround to fix this issue is to re-encode the private key using the OpenSSL tool: D:\> rename crashapache2.key crashapache2-java.key D:\> D:\Apache2\bin\openssl rsa -in crashapache2-java.key -out crashapache2.key Then use the re-encoded crashapache2.key to add to the crashapache2.crtkey file. REMINDER: The crashapache2.crtkey file must have UNIX-style line endings! After restarting Apache, it should be able to access the remote server (Tomcat in my case) without any problems.
Created attachment 13197 [details] Tomcat 5.0.28 config file
Created attachment 13198 [details] ExportPriv.java: source for the ExportPriv class
Created attachment 13199 [details] httpd.conf: Apache 2.0.52 config file
Created attachment 13200 [details] 31856.jks: Java keystore holding example client cert and private key
Created attachment 13201 [details] crashapache2-bad.crtkey: PEM-encoded cert and private key (causes crash)
Created attachment 13202 [details] crashapache2.crtkey: PEM-encoded cert and private key (works)
Test environment setup: 1. To set up an instance of Tomcat 5.0.28, download the Tomcat 5.0.28 config file (rename to "server.xml" and replace the default server.xml), and create a self-signed certificate and key called keystore.jks (password: pass00) using the Java keytool. 2. Download the Apache config file, rename to "httpd.conf", and replace the default httpd.conf file. You will have to change the following items in this file: - "W.X.Y.Z" with a local IP address to which to bind - "local.host.name" with the local hostname for the "W.X.Y.Z" IP address - "remote.host.name" with the remote hostname (to which Tomcat is bound) You will also have to create a certificate and a private key called "local-ssl.crt" and "local-ssl.key" to run SSL on Apache (or optionally disable these directives). 3. Download 31856.jks and place it in Tomcat's conf directory. 4. Download crashapache2-bad.crtky and crashapache2.crtkey and place them in Apache's conf directory. Rename crashapache2-bad.crtkey to crashapache2.crtkey to test the crash condition. 5. Restart both servers and test. Note that you may also use OpenSSL to create a pkcs12 file from the certificate and the private key (see http://mark.foster.cc/kb/openssl-keytool.html) to test hitting Tomcat directly from a web browser. Simply import the pkcs12 file into "Your Certificates" in Mozilla or Firefox to test this. Whew! I hope that's enough info to reproduce this issue.
I had a thought on the way home as to what the problem may be. If mod_ssl assumes (requires) that the base64-encoded private key end with an equals sign (=), then the bad key could be fixed by adding an equals sign to the end of the base64-encoded text. (Conversely, the good key could be made bad by removing the equals sign.) This should be fairly easy to verify (through testing or through code review).
1) What gets logged with "LogLevel debug", particularly the "Proxy client certificate callback:" calls? 2) can you apply the patch referenced from bug 24030, rebuild and retest? It doesn't look like the same problem but it would be quick to eliminate. 3) what version of OpenSSL
1) LogLevel debug shows the following just before the child process is restarted: [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 read server certificate A [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 read server key exchange A [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 read server certificate request A [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1764): OpenSSL: Loop: SSLv3 read server done A [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1530): Proxy client certificate callback: (web.server.name:443) entered [Sat Oct 23 14:24:06 2004] [debug] ssl_engine_kernel.c(1503): Proxy client certificate callback: (web.server.name:443) found acceptable cert, sending /C=US/ST=State/L=City/O=Organization/OU=Unit/CN=common-name After that, the log outputs startup (init) information, beginning with: [Sat Oct 23 14:24:10 2004] [info] Loading certificate & private key of SSL-aware server [Sat Oct 23 14:24:10 2004] [debug] ssl_engine_pphrase.c(468): unencrypted RSA private key - pass phrase not required [Sat Oct 23 14:24:11 2004] [info] Configuring server for SSL protocol 2) Bug 24030 was marked as resolved on May 25, 2004. Surely that code has been included in Apache-2.0.52 by now? Also, I'm using a pre-built binary, so I am not set up to build my own distribution of Apache (see Comment #1). 3) As mentioned in Comment #1, I'm using a pre-built binary of Apache-2.0.52 and OpenSSL-0.9.7d.
Upon further investigation by Mark Foster (http://mark.foster.cc/), Java was exporting the private key in PKCS#8 format, not RSA format, which is why the MIME text of the private key doesn't match between the crashapache2-bad.crtkey and the crashapache2.crtkey files. That means that this bug is really a case where using a private key in PKCS#8 format causes Apache to crash with the SSLProxyMachineCertificateFile directive. Yes, it was due to ignorance on my part, but I figured this still may be fixable. Mark explains (in an email to me): I noticed something odd about your exported private key. It has -----BEGIN PRIVATE KEY----- and not -----BEGIN RSA PRIVATE KEY----- which got me to poking around and lead me to this http://www.openssl.org/docs/apps/pkcs8.html Sure enough, this works... openssl pkcs8 -inform PEM -nocrypt -in exported.key -out exported4.pem In other words the output of this command looks exactly like the rsa. So the private key is being exported as PKCS#8 PEM format. It's not corrupted per se. Apache must be expecting an RSA key in the cert/key file. -- Some days it's just not worth chewing through the restraints... Mark D. Foster, CISSP <mark@foster.cc> http://mark.foster.cc/
The fix for bug 24030 is proposed for inclusion in 2.0.53, but is not in 2.0.52. It is the same bug, anyway; mod_ssl refuses to start up with the bogus keypair configured in HEAD: incomplete client cert configured for SSL proxy (missing or encrypted private key?) *** This bug has been marked as a duplicate of 24030 ***