[YARN-7644] NM gets backed up deleting docker containers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Component/s: nodemanager
• Labels: Docker
• Hadoop Flags: Reviewed
• Assignee: Chandni Singh
• Reporter: Eric Badger
• Created: 12/Dec/17 19:04
• Updated: 30/Nov/18 09:50
• Resolved: 10/Oct/18 15:07

Description

We are sending a docker stop to the docker container with a timeout of 10 seconds when we shut down a container. If the container does not stop after 10 seconds then we force kill it. However, the docker stop command is a blocking call. So in cases where lots of containers don't go down with the initial SIGTERM, we have to wait 10+ seconds for the docker stop to return. This ties up the ContainerLaunch handler and so these kill events back up. It also appears to be backing up new container launches as well.

Issue Links

• is related to: YARN-9074 Docker container rm command should be executed after stop
• relates to: YARN-8706 DelayedProcessKiller is executed for Docker containers even though docker stop sends a KILL signal after the specified grace period