[IGNITE-14324] EVT_CLIENT_NODE_DISCONNECTED is not triggered in k8s

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.1
• Fix Version/s: 2.11
• Labels: discovery
• Release Note: Fixed an issue that prevented a node from startup in case it can't obtain IP addresses to connect to
• Ignite Flags: Release Notes Required
• Assignee: Alexandr Shapkin
• Reporter: Alexandr Shapkin
• Created: 16/Mar/21 12:11
• Updated: 19/Apr/21 09:41
• Resolved: 19/Apr/21 07:16

Description

Scenario:

Kubernetes world, a server node, a k8s service, and a thick client. The client is subscribed to EVT_CLIENT_NODE_DISCONNECTED event and is connected to the server node.

When a service goes down alongside the server, no EVT_CLIENT_NODE_DISCONNECTED is caught and the client reports only about a socket exception and inability to resolve IP addresses from the services, cause k8s resource is not available. The DEBUG logs (attached) show that discovery is constantly trying to use KubernetesIpFinder and restore the connection.

Expected:
Discovery realizes that a client is disconnected, no more reconnection attempts happen and EVT_CLIENT_NODE_DISCONNECTED is thrown.

Solution:

Count resolution attempts and if it’s more than the threshold (2 if failuredetectiontimeout is not configured, otherwise #reconnCount) give up and invoke disconnection logic

Issue Links

• links to: GitHub Pull Request #8980