[FLINK-21053] Prevent potential RejectedExecutionExceptions in CheckpointCoordinator failing JM

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Fix Version/s: 1.20.0
• Component/s: Runtime / Checkpointing
• Labels: auto-unassigned
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 20/Jan/21 09:56
• Updated: 11/Mar/24 12:44

Description

In the past, there were multiple bugs caused by throwing/handling RejectedExecutionException in CheckpointCoordinator (FLINK-18290, FLINK-20992).

 

And I think it's still possible as there are many places where an executor is passed to calls to CompletableFuture.xxxAsync while it can already be shut down.

 

In FLINK-20992 we discussed two approaches to fix this.

One approach is to check executor state inside a synchronized block every time when it is used.

Second approach is to

1. Create executors inside CheckpointCoordinator (both io & timer thread pools)
2. Check isShutdown() in their RejectedExecution handlers (if yes and it's RejectedExecutionException then just log; otherwise delegate to FatalExitExceptionHandler)
3. (this will allow to remove such RejectedExecutionException checks from coordinator code)

 

Issue Links

• relates to: FLINK-20992 Checkpoint cleanup can kill JobMaster