I am new to Tomcat and to this mailing list. Looked far and wide for a solution to my problem, but couldn't find anything effective. I found other folks asking about similar issues. I then looked through the source and think I got a solution that I'd like to share as a patch. The problem is this: Trying to use Apache Tomcat with an OpenSSL Engine that has proprietary private ECC key format fails. The private key file is not PEM, and only this specific OpenSSL Engine can load such a private ECC key. When the server.xml configuration includes reference to a proprietary format private ECC key, in a Service/Connector/SSLHostConfig/Certificate/certificateKeyFile, the run-time fails to initialize a new SSL context. As a result, TLS doesn't get established, connection fails. I have tried Tomcat7, 9 and 10. To illustrate the configuration in server.xml, it includes elements like these: <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="MySslEngine" /> <Service name="Catalina"> <!-- Define an SSL Coyote HTTP/1.1 Connector on port 8443 --> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol" port="8443" maxThreads="200" scheme="https" SSLEnabled="true"> <SSLHostConfig caCertificateFile="my-keys/ca.pem" certificateVerification="require" certificateVerificationDepth="10"> <Certificate certificateKeyFile="my-keys/server.key" certificateFile="my-keys/server.pem" certificateChainFile="my-keys/server-chain.pem" type="EC" /> </SSLHostConfig> </Connector> <!-- ... --> </Service> The logs may include lines like these: 05-Mar-2021 14:37:07.175 INFO [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [/opt/my-keys/server.pem] or its private key [/opt/my-keys/server.key] could not be processed using a JSSE key manager and will be given directly to OpenSSL 05-Mar-2021 14:37:07.176 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context java.lang.Exception: Unable to load certificate key /opt/my-keys/server.key (error:0909006C:PEM routines:get_name:no start line) at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method) at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379) at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250) at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:401) at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:367) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1164) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1177) at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:574) at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:82) at org.apache.catalina.connector.Connector.initInternal(Connector.java:1052) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardService.initInternal(StandardService.java:558) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1045) at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136) at org.apache.catalina.startup.Catalina.load(Catalina.java:747) at org.apache.catalina.startup.Catalina.load(Catalina.java:769) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:302) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:472) My understanding of the root cause is that Tomcat doesn't support a proprietary format of the private ECC key. It insists that the key be in PEM format, in a file or in a keystore. What I needed was support for the "engine" key format. Similar to the feature of the "openssl digest" command in the following invocation: openssl dgst \ -sign my-keys/server.key \ -keyform ENGINE \ -engine MySslEngine \ -out signature.bin \ my-input When the key has the form "engine", the key is loaded using the ENGINE_load_private_key API (https://www.openssl.org/docs/man1.1.0/man3/ENGINE_load_private_key.html). I have come up with a small change to the Tomcat Native library that resolves the problem for me. It is not as general as the "engine" key form in the openssl command line. The change below simply attempts to load the private key through the ENGINE_load_private_key if load_pem_key fails. Please consider the change as a patch to the Tomcat Native library: --- tomcat-native-1.2.26-src/native/include/ssl_private.h 2020-12-10 09:09:19.000000000 -0800 +++ tomcat-native-1.2.26-src.changed/native/include/ssl_private.h 2021-03-08 15:35:56.516968836 -0800 @@ -51,6 +51,7 @@ */ #ifndef OPENSSL_NO_ENGINE #include <openssl/engine.h> +extern ENGINE *tcn_ssl_engine; #endif #ifndef RAND_MAX --- tomcat-native-1.2.26-src/native/src/sslcontext.c 2020-12-10 09:09:19.000000000 -0800 +++ tomcat-native-1.2.26-src.changed/native/src/sslcontext.c 2021-03-08 15:38:18.264651853 -0800 @@ -1034,7 +1034,13 @@ } } else { - if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL) { + if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL +#ifndef OPENSSL_NO_ENGINE + && tcn_ssl_engine && + (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file, + NULL, NULL)) == NULL +#endif + ) { ERR_error_string(SSL_ERR_get(), err); tcn_Throw(e, "Unable to load certificate key %s (%s)", key_file, err);
Changed Importance to P1 / Major as this blocks use of OpenSSL Engine with TPMs or HSMs, and OpenSSL Engine support is included in Tomcat Native library already.
Changing back to "enhancement" since that's what this is. It may be of major importance to YOU but this is _not_ a bug.
A quick review: > ... > && tcn_ssl_engine && > (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file, > NULL, NULL)) == NULL This will attempt to use the tcn_ssl_engine pointer to load the private key, but only if the pointer is non-zero. If you mean NULL, you should explicitly compare the pointer to NULL. Otherwise the patch looks reasonable to me. I assume you've tested it to make sure it still works with a "standard" configuration?
Thanks for the updates, the review and the comments. The NULL comparison I can add. The change was tested with and without the SSL engine in the configuration. Is there anything else I can do to get the patch included in the source? Could I submit a pull request somehow?
(In reply to Edin Hodzic from comment #4) > Could I submit a pull request somehow? https://github.com/apache/tomcat-native
Thanks for the patch. The fix will be in 1.2.27. The Tomcat 10.0.x unit tests passed for the APR/Native connector with this patch applied.
Thanks everyone for the prompt action and the pending change.
Looking at commit, that implemented this feature for Tomcat Native 1.2.27, 1. There is a bug. https://github.com/apache/tomcat-native/commit/69e884a96a308a2bfdd91c7de3a9b301838031c8 in native/src/sslcontext.c, lines 1039-1040 > - if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL) { > + if ((c->keys[idx] = load_pem_key(c, key_file)) == NULL > +#ifndef OPENSSL_NO_ENGINE > + && tcn_ssl_engine != NULL && > + (c->keys[idx] = ENGINE_load_private_key(tcn_ssl_engine, key_file, > + NULL, NULL)) == NULL > +#endif > + ) { I think that when one uses the default configuration (when there is no custom engine configured with SSLEngine attribute of AprLifecycleListener) and thus tcn_ssl_engine is NULL, the behaviour should remain unchanged. As such, the code above has to be "&& (tcn_ssl_engine == NULL || ENGINE_load_private_key() fails)". With the current code when load_pem_key call fails and tcn_ssl_engine == NULL the error reporting block is skipped. 2. I am OK with the patch with the above bug fixed, but for future improvements there are some thoughts: 1) I wonder, whether in case of a custom engine (tcn_ssl_engine != NULL) it would be better to try a "ENGINE_load_private_key" call first. 2) As "load_pem_key" is our own method (implemented in the same sslcontext.c file), I wonder whether it would be better to move the change there (and maybe rename that method). 3) I see that "ENGINE_load_private_key" method will be deprecated in OpenSSL 3.0 (master) https://www.openssl.org/docs/man1.1.1/man3/ENGINE_load_private_key.html https://www.openssl.org/docs/manmaster/man3/ENGINE_load_private_key.html 4) How about a "ENGINE_load_public_key" method? There is no need to call it? STEPS TO REPRODUCE (for 1. bug) Using release candidate of Tomcat 9.0.45: 1) Install Tomcat, e.g. unzip apache-tomcat-9.0.45-windows-x64.zip on Windows. 2) Edit the conf/server.xml file. - ~line 68: Commend out the default HTTP connector on port 8080 - ~line 102: Uncomment a connector that uses OpenSSL for TLS 3) Create two files with some dummy content (e.g. "foo bar"): conf/localhost-rsa-cert.pem conf/localhost-rsa-key.pem 4) Start Tomcat. ACTUAL BEHAVIOUR: With Tomcat 9.0.45 and Tomcat Native 1.2.27 a startup failure is reported as follows: [[[ 30-Mar-2021 18:28:20.133 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context java.lang.Exception: Unable to load certificate [censored]\conf/localhost-rsa-cert.pem (error:0D06B08E:asn1 encoding routines:asn1_d2i_read_bio:not enough data) at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method) at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379) at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250) at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:397) at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:363) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1204) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1217) ]]] EXPECTED BEHAVIOUR: If I replace bin/tcnative-1.dll file with Tomcat Native 1.2.26 (e.g. the one from Tomcat 9.0.44), the error is reported as follows: [[[ 30-Mar-2021 18:30:02.551 WARNING [main] org.apache.tomcat.util.net.openssl.OpenSSLContext.init Error initializing SSL context java.lang.Exception: Unable to load certificate key [censored]\conf/localhost-rsa-key.pem (error:0909006C:PEM routines:get_name:no start line) at org.apache.tomcat.jni.SSLContext.setCertificate(Native Method) at org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(OpenSSLContext.java:379) at org.apache.tomcat.util.net.openssl.OpenSSLContext.init(OpenSSLContext.java:250) at org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(SSLUtilBase.java:246) at org.apache.tomcat.util.net.AprEndpoint.createSSLContext(AprEndpoint.java:397) at org.apache.tomcat.util.net.AprEndpoint.bind(AprEndpoint.java:363) at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1204) at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1217) ]]] As loading the key in SSLContext::setCertificate (in native/src/sslcontext.c) happens earlier than loading the certificate, the expected error message is "Unable to load certificate key". Seeing the "Unable to load certificate" message means that error handling for the former was skipped.
> As such, the code above has to be "&& (tcn_ssl_engine == NULL || ENGINE_load_private_key() fails)". I agree, my bad on the original patch. :/
Fixed for 1.2.28 onwards.
Created attachment 37870 [details] keep structural reference to the SSL Engine pointed to by tcn_ssl_engine Found another problem with the recent engine related change. Got to keep a "structural" reference to the engine so the tcn_ssl_engine pointer remains valid at the time it's used for loading private key. Please see the tcn-1.2.28.diff attachment for a patch.
Comment on attachment 37870 [details] keep structural reference to the SSL Engine pointed to by tcn_ssl_engine diff -aur tomcat-native-1.2.28-src/native/src/ssl.c tomcat-native-1.2.28-src.changed/native/src/ssl.c --- tomcat-native-1.2.28-src/native/src/ssl.c 2021-04-01 06:28:54.000000000 -0700 +++ tomcat-native-1.2.28-src.changed/native/src/ssl.c 2021-05-20 13:21:18.569612200 -0700 @@ -367,6 +367,14 @@ #endif free_dh_params(); +#ifndef OPENSSL_NO_ENGINE + if (tcn_ssl_engine != NULL) { + /* Release the SSL Engine structural reference */ + ENGINE_free(tcn_ssl_engine); + tcn_ssl_engine = NULL; + } +#endif + #if OPENSSL_VERSION_NUMBER >= 0x10100000L && !defined(LIBRESSL_VERSION_NUMBER) /* Openssl v1.1+ handles all termination automatically. Do * nothing in this case. @@ -815,9 +823,9 @@ if (!ENGINE_set_default(ee, ENGINE_METHOD_ALL)) err = APR_ENOTIMPL; } - /* Free our "structural" reference. */ + /* Keep our "structural" reference. */ if (ee) - ENGINE_free(ee); + tcn_ssl_engine = ee; } if (err != APR_SUCCESS) { TCN_FREE_CSTRING(engine); @@ -825,7 +833,6 @@ tcn_ThrowAPRException(e, err); return (jint)err; } - tcn_ssl_engine = ee; } #endif
Created attachment 37871 [details] keep structural reference to the SSL Engine pointed to by tcn_ssl_engine improved diff
One other potential improvement in the OpenSSL interaction might be to enable OpenSSL configuration: In ssl.c, initialization is done with the call: OPENSSL_init_ssl(OPENSSL_INIT_ENGINE_ALL_BUILTIN, NULL); The openssl command in OpenSSL source performs initialization like this: OPENSSL_init_ssl(OPENSSL_INIT_ENGINE_ALL_BUILTIN | OPENSSL_INIT_LOAD_CONFIG, NULL); The OPENSSL_INIT_LOAD_CONFIG makes the OpenSSL library load configuration (for example /etc/ssl/openssl.cnf by default on Ubuntu). The configuration could also be specified through a path in OPENSSL_CONF environment variable (that could be set in setenv.sh in Tomcat for full control over the OpenSSL configuration.)
Mark, this has been marked as fixed in the changelog for 1.2.28. This appears weird to the user. We should settle this timely or create a followup ticket.
I've applied a variation of the additional fix in comment #12. The suggestion in #14 is a separate issue and should be submitted via a new enhancement request.