[FLINK-19171] K8s Resource Manager may lead to resource leak after pod deleted

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Assignee: Unassigned
• Reporter: Yi Tang
• Created: 09/Sep/20 07:29
• Updated: 10/Sep/20 11:19
• Resolved: 10/Sep/20 11:19

Description

private void terminatedPodsInMainThread(List<KubernetesPod> pods) {
   getMainThreadExecutor().execute(() -> {
      for (KubernetesPod pod : pods) {
         if (pod.isTerminated()) {
            ...
         }
      }
   });
}

Looks like that the RM only remove the pod from ledger if the pod "isTerminated", 

and the pod has been taken accounted after being created.

However, it is not complete by checking pod "isTerminated", e.g. a Pending pod is deleted manually.

After that, a new job requires more resource can not trigger the allocation of a new pod.

 

Pls let me know if i misunderstand, thanks.

Issue Links

• relates to: FLINK-13554 ResourceManager should have a timeout on starting new TaskExecutors.
• relates to: FLINK-18229 Pending worker requests should be properly cleared