[GEODE-9910] Failure to auto-reconnect upon network partition

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.12.10, 1.13.9, 1.14.5, 1.15.0
• Labels: GeodeOperationAPI blocks-1.15.0​ needsTriage pull-request-available
• Assignee: Barrett Oglesby
• Reporter: Surya Mudundi
• Created: 03/Jan/22 16:09
• Updated: 24/Jun/22 22:25
• Resolved: 11/Mar/22 00:54

Description

Two node cluster with embedded locators failed to auto-reconnect when node-1 experienced network outage for couple of minutes and when node-1 recovered from the outage, node-2 failed to auto-reconnect.

node-2 tried to re-connect to node-1 as:

[org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] Attempting to reconnect to the distributed system.  This is attempt #1.
[org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] Attempting to reconnect to the distributed system.  This is attempt #2.
[org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] Attempting to reconnect to the distributed system.  This is attempt #3.

Finally reported below error after 3 attempts as:

INFO  [org.apache.geode.logging.internal.LoggingProviderLoader]-[ReconnectThread] [] Using org.apache.geode.logging.internal.SimpleLoggingProvider for service org.apache.geode.logging.internal.spi.LoggingProvider
INFO  [org.apache.geode.internal.InternalDataSerializer]-[ReconnectThread] [] initializing InternalDataSerializer with 0 services
INFO  [org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] performing a quorum check to see if location services can be started early
INFO  [org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] Quorum check passed - allowing location services to start early
WARN  [org.apache.geode.distributed.internal.InternalDistributedSystem]-[ReconnectThread] [] Exception occurred while trying to connect the system during reconnect
java.lang.IllegalStateException: A locator can not be created because one already exists in this JVM.
        at org.apache.geode.distributed.internal.InternalLocator.createLocator(InternalLocator.java:298) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalLocator.createLocator(InternalLocator.java:273) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.startInitLocator(InternalDistributedSystem.java:916) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:768) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.access$200(InternalDistributedSystem.java:135) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem$Builder.build(InternalDistributedSystem.java:3034) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.connectInternal(InternalDistributedSystem.java:290) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2605) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2424) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1275) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:2326) ~[geode-core-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership.uncleanShutdown(GMSMembership.java:1187) ~[geode-membership-1.14.0.jar:?]
        at org.apache.geode.distributed.internal.membership.gms.GMSMembership$ManagerImpl.lambda$forceDisconnect$0(GMSMembership.java:1811) ~[geode-membership-1.14.0.jar:?]
        at java.lang.Thread.run(Thread.java:829) [?:?]

 

Issue Links

• links to: GitHub Pull Request #7393