[YARN-10341] Yarn Service Container Completed event doesn't get processed

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.1
• Fix Version/s: 3.4.0, 3.3.1
• Component/s: service-scheduler
• Target Version/s: 3.4.0, 3.3.1
• Hadoop Flags: Reviewed
• Assignee: Bilwa S T
• Reporter: Bilwa S T
• Created: 06/Jul/20 17:19
• Updated: 11/Feb/24 12:50
• Resolved: 09/Jul/20 07:12

Description

If there 10 workers running and if containers get killed , after a while we see that there are just 9 workers runnning. This is due to CONTAINER COMPLETED Event is not processed on AM side.
Issue is in below code:

public void onContainersCompleted(List<ContainerStatus> statuses) {
      for (ContainerStatus status : statuses) {
        ContainerId containerId = status.getContainerId();
        ComponentInstance instance = liveInstances.get(status.getContainerId());
        if (instance == null) {
          LOG.warn(
              "Container {} Completed. No component instance exists. exitStatus={}. diagnostics={} ",
              containerId, status.getExitStatus(), status.getDiagnostics());
          return;
        }
        ComponentEvent event =
            new ComponentEvent(instance.getCompName(), CONTAINER_COMPLETED)
                .setStatus(status).setInstance(instance)
                .setContainerId(containerId);
        dispatcher.getEventHandler().handle(event);
      }

If component instance doesnt exist for a container, it doesnt iterate over other containers as its returning from method. This happens when restart_policy is "ON_FAILURE"