[FLINK-16511] Task cancellation timeout is not effective on OOM errors

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Runtime / Task
• Assignee: Maximilian Michels
• Reporter: Maximilian Michels
• Created: 09/Mar/20 20:05
• Updated: 10/Mar/20 17:55
• Resolved: 10/Mar/20 17:55

Description

Under high memory pressure, the task manager shutdown on fatal errors is not reliable:

If a task does not cooperate and cannot be canceled and there is a OOM when starting the task cancellation watchdog thread, the exception is not propagated correctly. The reason for this is that the job manager retries the cancelTask() request multiple times. The operation is stateful and if we fail to start the watchdog thread, we won't attempt it again as the task already switches to the CANCELING state before starting the watchdog thread.

Such fatal errors should automatically shutdown the task manager without a retry form the job manager side.

Issue Links

• causes: FLINK-10850 Job may hang on FAILING state if taskmanager updateTaskExecutionState failed
• duplicates: FLINK-14949 Task cancellation can be stuck against out-of-thread error
• relates to: FLINK-16510 Task manager safeguard shutdown may not be reliable