[FLINK-21902] A deadlock while using K8s HA service

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Yi Tang
• Created: 22/Mar/21 03:30
• Updated: 22/Mar/21 06:56
• Resolved: 22/Mar/21 06:56

Description

The `KubernetesStateHandleStore` using the same threadPoolExecutor with the Dispatcher to check `checkAndUpdateConfigMap`, which will lead to a deadlock.

example:

 private CompletableFuture<Void> removeJob(JobID jobId, CleanupJobState cleanupJobState)

{ final DispatcherJob job = checkNotNull(runningJobs.remove(jobId)); final CompletableFuture<Void> jobTerminationFuture = job.closeAsync(); return jobTerminationFuture.thenRunAsync( () -> cleanUpJobData(jobId, cleanupJobState.cleanupHAData), ioExecutor); }

will finally call

 public CompletableFuture<Boolean> checkAndUpdateConfigMap(
 String configMapName,
 Function<KubernetesConfigMap, Optional<KubernetesConfigMap>> function) {
 ...
 CompletableFuture.supplyAsync(..., kubeClientExecutorService)
 ...
 }

And the ioExecutor and kubeClientExecutorService  is the same executor.

Issue Links

• is duplicated by: FLINK-21685 Flink JobManager failed to restart from checkpoint in kubernetes HA setup