[FLINK-23209] Timeout heartbeat if the heartbeat target is no longer reachable

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.1, 1.12.4
• Fix Version/s: 1.14.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

Flink now supports to detect dead TaskManagers via the number of consecutive failed heartbeat RPCs. The threshold until a TaskManager is marked as unreachable can be configured via `heartbeat.rpc-failure-threshold`. This can speed up the detection of dead TaskManagers significantly.

• Assignee: Till Rohrmann
• Reporter: Till Rohrmann
• Created: 01/Jul/21 13:24
• Updated: 14/Dec/21 09:06
• Resolved: 16/Jul/21 07:15

Description

With FLINK-23202 it should now be possible to see when a remote RPC endpoint is no longer reachable. This can be used by the HeartbeatManager to mark an heartbeat target as no longer reachable. That way, it is possible for Flink to react faster to losses of components w/o having to wait for the heartbeat timeout to expire. This will result in faster recoveries (e.g. if a TaskExecutor dies).

With this change we can improve trading off speed of detecting dead TaskManagers against running on an unstable/overloaded network where heartbeat messages are delayed.

Issue Links

• depends upon: FLINK-23202 RpcService should fail result futures if messages could not be sent
• links to: GitHub Pull Request #16357