[DISPATCH-2262] Edge/Interior connections can half-fail in real multi-cloud environments

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.17.0
• Fix Version/s: 1.18.0
• Component/s: Container
• Assignee: Ted Ross
• Reporter: Ted Ross
• Created: 26/Oct/21 14:45
• Updated: 19/Nov/21 21:43
• Resolved: 03/Nov/21 19:07

Description

See PROTON-2440 for context.

The configured keepalive on edge-to-interior connections can fail to provide protection from connection loss. This results in half-failed connections where the edge sees connection failure and re-connects and the interior sees nothing and accumulates multiple connections from the same edge.

This is a serious problem because the interior will attempt to forward deliveries across these dead-but-seemingly-alive connections resulting in lack of message delivery.

The solution proposed in this issue is to work around the problem by introducing an application-level keepalive for edge-to-interior connections.

Issue Links

• fixes: DISPATCH-2283 heap-use-after-free in system_tests_policy_oversize_compound during qdrc_endpoint_delivery_CT
• fixes: DISPATCH-2117 Assertion failure outs.count("edge-downlink") != 4 in system_tests_edge_router
• is related to: PROTON-2440 Configured heartbeats sometimes don't function in inter-cloud environments