[GEODE-4451] When a gateway sender is created using gfsh and fails to connect due to AuthenticationFailedException, it is left in an inconsistent state

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: wan
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Barrett Oglesby
• Created: 01/Feb/18 01:59
• Updated: 02/May/18 18:21
• Resolved: 22/Mar/18 16:44

Description

If I attempt to create a gateway sender with invalid credentials using gfsh, it fails like:

(2) Executing - create gateway-sender --remote-distributed-system-id=1 --id=ny --dispatcher-threads=1 --parallel=true

Member | Status
------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ln-1 | ERROR: Could not start a gateway sender ny because of exception Could not start a gateway sender 84 because of exception org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.

The 84 should be the gateway sender id; instead its the thread id (from ConcurrentParallelGatewaySenderEventProcessor.waitForRunningStatus.

In the server log, I see 5 AuthenticationFailedException stacks like (one per dispatcher-thread):

[severe 2018/01/31 16:13:12.638 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x55] Message dispatch failed due to unexpected exception..
org.apache.geode.internal.cache.wan.GatewaySenderException: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials., caused by org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:432)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.<init>(GatewaySenderEventRemoteDispatcher.java:75)
 at org.apache.geode.internal.cache.wan.parallel.RemoteParallelGatewaySenderEventProcessor.initializeEventDispatcher(RemoteParallelGatewaySenderEventProcessor.java:74)
 at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.setRunningStatus(AbstractGatewaySenderEventProcessor.java:1078)
 at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1050)
Caused by: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
 at org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
 at org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
 at org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
 at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
 at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
 at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
 at org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:388)
 ... 4 more

A thread dump shows the Event Processor threads are gone, but 5 BatchRemovalThreads like this exist:

"BatchRemovalThread" #96 daemon prio=10 os_prio=31 tid=0x00007fc1dd009000 nid=0xcb03 waiting on condition [0x000070000e245000]
 java.lang.Thread.State: TIMED_WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x00000007a04043c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
 at org.apache.geode.internal.util.concurrent.StoppableCondition.await(StoppableCondition.java:61)
 at org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue$BatchRemovalThread.run(ParallelGatewaySenderQueue.java:1613)

list gateways shows the sender in the Not Running state:

GatewaySender Id | Member | Remote Cluster Id | Type | Status | Queued Events | Receiver Location
---------------- | -------------------------------- | ----------------- | -------- | ----------- | ------------- | -----------------
ny | 192.168.2.4(ln-1:13883)<v3>:1027 | 1 | Parallel | Not Running | 0 | null

The sender is left in an inconsistent state. It should either be completely gone or retrying to connect.

GatewaySenderEventRemoteDispatcher.<init> catches the GatewaySenderException attempting to initializeConnection and rethrows it if is caused by GemFireSecurityException:

public GatewaySenderEventRemoteDispatcher(AbstractGatewaySenderEventProcessor eventProcessor) {
 this.processor = eventProcessor;
 this.sender = eventProcessor.getSender();
 // this.ackReaderThread = new AckReaderThread(sender);
 try {
 initializeConnection();
 } catch (GatewaySenderException e) {
 if (e.getCause() instanceof GemFireSecurityException) {
 throw e;
 }
 }
}

AbstractGatewaySenderEventProcessor.run catches that exception and logs severe message and stack above.

If I change this area of code to log the warning instead of stopping the Event Processor threads, I see this exception after puts are done:

[severe 2018/01/31 17:37:16.324 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x49] Stopping the processor because the following exception occurred while processing a batch:
org.apache.geode.internal.cache.wan.GatewaySenderException: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials., caused by org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:431)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.getConnection(GatewaySenderEventRemoteDispatcher.java:320)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher._dispatchBatch(GatewaySenderEventRemoteDispatcher.java:198)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.dispatchBatch(GatewaySenderEventRemoteDispatcher.java:147)
 at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:609)
 at org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1051)
Caused by: org.apache.geode.security.AuthenticationFailedException: Authentication error. Please check your credentials.
 at org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397)
 at org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253)
 at org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118)
 at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136)
 at org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259)
 at org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206)
 at org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902)
 at org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:387)
 ... 5 more

That is coming from GatewaySenderEventRemoteDispatcher.dispatchBatch:

} catch (GatewaySenderException ge) {
 if (...) {
 ...
 } else {
 logger.fatal(
 LocalizedMessage.create(
 LocalizedStrings.GatewayEventRemoteDispatcher_STOPPING_THE_PROCESSOR_BECAUSE_THE_FOLLOWING_EXCEPTION_OCCURRED_WHILE_PROCESSING_A_BATCH),
 ge);
 this.processor.setIsStopped(true);
 }

So, AuthenticationFailedExceptions are definitely meant to stop the sender.

Issue Links

• links to: GitHub Pull Request #1606