[FLINK-23202] RpcService should fail result futures if messages could not be sent

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.1, 1.12.4
• Fix Version/s: 1.14.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

The same way Flink detects unreachable heartbeat targets faster, Flink now also immediately fails RPCs where the target is known by the OS to be unreachable on a network level, instead of waiting for a timeout (akka.ask.timeout).

One effect this are faster task failovers, because cancelling tasks on a dead TaskExecutor no longer gets delayed by the RPC timeout.

If this faster failover is a problem in certain setups (which might rely on the fast that external systems hit timeouts), we recommend to configure the application's restart strategy with a restart delay.

• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 01/Jul/21 08:48
• Updated: 10/Sep/21 07:54
• Resolved: 08/Jul/21 07:21

Description

The RpcService should fail result futures if messages could not be sent. This would speed up the failure detection mechanism because it would not rely on the timeout. One way to achieve this could be to listen to the dead letters and then sending a Failure message back to the sender.

Issue Links

• is depended upon by: FLINK-23209 Timeout heartbeat if the heartbeat target is no longer reachable
• is related to: FLINK-23318 AkkaRpcActorTest#testOnStopFutureCompletionDirectlyTerminatesAkkaRpcActor fails on azure
• links to: GitHub Pull Request #16342