[GEODE-10066] SSL handshake failures on 1 locator prevents connection pool from trying other locators

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.9, 1.13.8, 1.14.4, 1.15.0
• Fix Version/s: 1.15.0
• Component/s: client/server
• Labels: pull-request-available ssl
• Assignee: Jacob Barrett
• Reporter: Jacob Barrett
• Created: 19/Feb/22 00:07
• Updated: 22/Jun/22 20:46
• Resolved: 26/Apr/22 17:34

Description

If an SSLException is thrown when handshaking with a locator the exception is wrapped in an IllegalStateException that is not caught by the connection pool, the stack is blown, and no connections can be established. If not wrapped the connection pool will properly try the next locator.

The SSLExceptions are wrapped in at least TcpClient.getServerVersion() but other locations may exist in this path. This method throws IOException and the SSLExceptions extend IOExceptions so they should not be wrapped. It probably makes sense to split the concern of socket connection from determining the server version in TcpClient.getServerVersion().

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address 10.2.8.12 found
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
	at org.apache.geode.internal.net.SocketCreator.configureClientSSLSocket(SocketCreator.java:594)
	at org.apache.geode.internal.net.SCAdvancedSocketCreator.connect(SCAdvancedSocketCreator.java:83)
	at org.apache.geode.distributed.internal.tcpserver.ClusterSocketCreatorImpl.connect(ClusterSocketCreatorImpl.java:96)
	at org.apache.geode.distributed.internal.tcpserver.TcpClient.getServerVersion(TcpClient.java:246)
	at org.apache.geode.distributed.internal.tcpserver.TcpClient.requestToServer(TcpClient.java:151)
	at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryOneLocatorUsingConnection(AutoConnectionSourceImpl.java:227)
	at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryOneLocator(AutoConnectionSourceImpl.java:217)
	at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.queryLocators(AutoConnectionSourceImpl.java:264)
	at org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.findServer(AutoConnectionSourceImpl.java:176)
	at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:211)
	at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:196)
	at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:190)
	at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:282)
	at org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:940)
	at org.apache.geode.cache.wan.internal.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:464)
	at org.apache.geode.cache.wan.internal.GatewaySenderEventRemoteDispatcher.<init>(GatewaySenderEventRemoteDispatcher.java:105)
	at org.apache.geode.cache.wan.internal.parallel.RemoteParallelGatewaySenderEventProcessor.initializeEventDispatcher(RemoteParallelGatewaySenderEventProcessor.java:66)
	at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.setRunningStatus(AbstractGatewaySenderEventProcessor.java:1107)
	at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1081)
Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 10.2.8.12 found
	at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:168)
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:94)
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:462)
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:428)
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:209)
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
	at org.apache.geode.internal.net.filewatch.FileWatchingX509ExtendedTrustManager.checkServerTrusted(FileWatchingX509ExtendedTrustManager.java:130)
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
	... 26 more

Issue Links

• causes: GEODE-10113 CI: AutoConnectionSourceImplTest.queryLocatorsTriesNextLocatorOnSSLExceptions() FAILED
• is caused by: GEODE-7379 Break dependencies on CancelException class
• is caused by: GEODE-1793 Flaky: LocatorDUnitTest.testStartTwoLocatorsOneWithSSLAndTheOtherNonSSL
• is caused by: GEODE-420 locator ssl configuration
• is related to: GEODE-7917 Problem forming SSL connection in multisite setup
• links to: GitHub Pull Request #7390
• links to: GitHub Pull Request #7433
• links to: GitHub Pull Request #7436