I'm using tomcat 9.0.21 and tomcat-native 1.2.21 on top of CentOS 7.6.1810. Current issue is this one from catalina.out: ----- 13-Jun-2019 10:43:32.132 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.4.8]. 13-Jun-2019 10:43:32.135 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. 13-Jun-2019 10:43:32.135 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true] 13-Jun-2019 10:43:32.141 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.0.2k-fips 26 Jan 2017] 13-Jun-2019 10:43:32.518 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-2007"] 13-Jun-2019 10:43:32.545 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-apr-2008"] # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f989117c860, pid=23986, tid=0x00007f9910185700 # # JRE version: OpenJDK Runtime Environment (8.0_191-b12) (build 1.8.0_191-b12) # Java VM: OpenJDK 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libcrypto.so.10+0x168860] X509_STORE_set_flags+0x0 # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # An error report file with more information is saved as: # /opt/temp/java_error23986.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # ----- The crash log shows ----- Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.tomcat.jni.SSLContext.setCARevocation(JLjava/lang/String;Ljava/lang/String;)Z+0 j org.apache.tomcat.util.net.openssl.OpenSSLContext.addCertificate(Lorg/apache/tomcat/util/net/SSLHostConfigCertificate;)V+77 j org.apache.tomcat.util.net.openssl.OpenSSLContext.init([Ljavax/net/ssl/KeyManager;[Ljavax/net/ssl/TrustManager;Ljava/security/SecureRandom;)V+191 j org.apache.tomcat.util.net.SSLUtilBase.createSSLContext(Ljava/util/List;)Lorg/apache/tomcat/util/net/SSLContext;+16 j org.apache.tomcat.util.net.AprEndpoint.createSSLContext(Lorg/apache/tomcat/util/net/SSLHostConfig;)V+81 ----- Which is the code https://github.com/apache/tomcat-native/blob/7a20a500587dc92944dee05cee5b388ae8443842/native/src/sslcontext.c#L640 Connector configuration: ----- <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" SSLRandomSeed="builtin" /> <Connector allowTrace="false" URIEncoding="UTF-8" acceptorThreadCount="2" acceptCount="400" connectionTimeout="20000" maxKeepAliveRequests="100" keepAliveTimeout="5000" maxThreads="400" minSpareThreads="25" tcpNoDelay="true" processorCache="750" protocol="org.apache.coyote.http11.Http11AprProtocol" port="2008" scheme="https" secure="true" SSLEnabled="true"> <SSLHostConfig certificateVerification="optional" caCertificateFile="/certs/server/x509_trusted.crt" certificateRevocationListFile="/certs/server/x509_trusted.crl" protocols="all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.3" ciphers="ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256" honorCipherOrder="true" disableCompression="true"> <Certificate certificateFile="/certs/server/server.crt" certificateChainFile="/certs/server/server_ca_chain.crt" certificateKeyFile="/certs/server/server.key" certificateKeyPassword="abc" /> </SSLHostConfig> </Connector> ----- When I remove the line setting the revogation list, "certificateRevocationListFile=\"/certs/server/x509_trusted.crl\"", it just works. Same PEM file works in old tomcat-7.0.69 using also APR and tomcat native. Tried versions back until 9.0.17 with same issue. With 9.0.16 and 9.014 I have different issue. On tomcat 9.0.14 same setup worked but on shutdown I had a core dump. With tomcat 9.0.16 same setup worked but on shutdown I had an even weird core dump. Starting tomcat 9.0.17 it stopped starting. I noticed that tomcat native changed from 1.2.19 to 1.2.21 when that happen. Tested also building a tomcat-native with latest openssl and latest apr as static linked in with same issue. So I assume it's not some weird library issue but something else. Happy to provide more data or even do other tests.
Looks like this fails on startup. Can you enable core dumps, re-try, and get a native stack trace of the error?
Created attachment 36623 [details] Error report generated by the JVM
Created attachment 36624 [details] Native stack trace from core dump of same process gdb java core where bt full
It does fail right at the beginning. Attached the traces.
Created attachment 36625 [details] Native stack trace from core dump of same process with debuginfo Installed the debug info and retrieved new native trace.
The stack trace suggests the fix for bug 62094 was incorrect. Looking at the trace, the proposed patch and the commit, I think the patch was not applied correctly.
Correcting product
I've dug into this some more. I see why the applied patch for bug 62094 differs from the proposed patch but the change made was incomplete. That was the cause of the crash observed here. Further testing found crashes on invalid CRL files and on shutdown. These have also been fixed. I have tested this with a CRL and both a revoked and non-revoked cert and everything appears to as expected.
Thank you for the investigation and fix. What is the release cycle for tomcat native and tomcat 9?
Current plans are: - 9.0.x early July - Native 1.2.22 before then so it can be packaged in the next 9.0.x release You can follow progress on the dev@ list.
Using the fix with tomcat 9.0.22 and tomcat-native 1.2.23 and the server doesn't core dump anymore when I add CRL to SSLHostConfig using configuration certificateRevocationListFile in format -----BEGIN X509 CRL----- But have a new issue, I'm unable to use valid X509 client certs. I get the message from curl curl: (35) error:1401E418:SSL routines:CONNECT_CR_FINISHED:tlsv1 alert unknown ca If I remove the CRL config client certs work as expected. Is there a way to understand what is the issue? Was unable to find a way to enable SSL logging being OpenSSL.