Using the tcnative library compiled against a FIPS-compatible build of OpenSSL on Windows results in a FIPS fingerprint error when the FIPSMode attribute in the AprLifecycleListener's Listener element in server.xml is set to "on". As a result, Tomcat fails to start. When the attribute is set to "off", no errors occur, Tomcat starts up successfully, and SSL services perform normally. The error that results in the catalina log file is: SEVERE: Failed to initialize the SSLEngine. java.lang.Exception: error:2D06B06F:FIPS routines:FIPS_check_incore_fingerprint:fingerprint does not match The problem appears to be that during startup, Tomcat's copy of the OpenSSL library libeay32.dll gets rebased from its desired memory address (by default, 0xFB00000) to a different address. Normally this wouldn't be an issue, but when operating in FIPS mode, the library executes a hash of itself which fails if it is not at its expected address. (See http://www.openssl.org/docs/fips/UserGuide-2.0.pdf, page 44.) This was determined by using Process Explorer to examine the tomcat7.exe process following a successful startup of Tomcat with FIPSMode set to "off". The libeay32.dll library in that process displayed an "Image Base" address of 0xFB00000, indicating its desired base memory address, an a "Base" address of 0x63E20000 (or something else on other servers), indicating the actual base memory address being used for the library. Additionally, the same procedure was used on a copy of the openssl.exe utility program, which successfully used FIPS mode with the same library used above. In that case, Process Explorer showed that the openssl.exe process' copy of libeay32.dll had the same value for the "Image Base" and "Base" addresses. Working around this problem involved recompiling the OpenSSL library to request a different base memory address. (I used 0x6FB00000.) Would it be possible to determine why, during startup, the libeay32.dll module's base memory address is being changed from the one it expects, and fix the problem? If not, it will need to be documented that in order to use a FIPS-compatible build of OpenSSL with TCNative in FIPS mode, the OpenSSL library will need to be recompiled with an alternate base memory address specified. (See the above PDF page for instructions for how to do so.) This issue was experienced using Tomcat 7.0.32, tcnative 1.1.27, APR 1.4.6, and OpenSSL 1.0.1c with FIPS module 2.0.4, all on Windows Server 2008.
I think this turns out to be a problem with the OpenSSL build, not the tcnative build. So, there's nothing that tcnative can do about it, especially because the ASF doesn't distribute a FIPS-enabled version of OpenSSL alongside tcnative (nor can they, as I understand). It is well worth noting in our build instructions that running with a FIPS-enabled library can run into these issues and the the solution is (I think) carefully building and linking OpenSSL such that all addresses are fixed.
You don't mention where you obtained the module, or what module collided with libeay32.dll load address, but if I had to guess, it has to do with the packager adopting a more modern build visual studio linker, and the bug is likely related to; http://markmail.org/search/?q=dev%40openssl.org+relocate+libeay32.dll#query:dev%40openssl.org%20relocate%20libeay32.dll+page:1+mid:jghhjvy4d7nhbipl+state:results I'm adding this just in case this might be true of the ASF distributed binary.
While Tomcat does provide a binary openssl.exe for Microsoft Windows users, it is not FIPS-enabled. The only way to get a "proper" FIPS-enabled binary is to build it yourself for your own uses. If we can find a procedure that will work with a modern MS Visual Studio, then we should certainly document it. I don't have access to a Windows toolchain, so I can't really be of any help.
We now have a documented and tested process for building a statically linked tc-native with a FIPS enabled OpenSSL on 64-bit Windows. I'm not concerned by the need to change the base address when building a dynamically linked library. The OpenSSL docs cover that.
Excellent. Thanks a bunch, Mark. Building with VS and OpenSSL is really no fun at all.