[YARN-8587] Delays are noticed to launch docker container

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.3.0
• Labels: Docker
• Target Version/s: 3.3.0, 3.2.1, 3.1.3
• Release Note: add "docker inspect" retries to discover container exit code.
• Assignee: dockerzhang
• Reporter: Yesha Vora
• Created: 26/Jul/18 18:08
• Updated: 19/Apr/19 19:42
• Resolved: 24/Oct/18 21:30

Description

Launch dshell application. Wait for application to go in RUNNING state.

yarn  jar /xx/hadoop-yarn-applications-distributedshell-*.jar  -shell_command "sleep 300" -num_containers 1 -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=httpd:0.1 -shell_env YARN_CONTAINER_RUNTIME_DOCKER_DELAYED_REMOVAL=true -jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell-xx.jar

Find out container allocation. Run docker inspect command for docker containers launched by app.

Sometimes, the container is allocated to NM but docker PID is not up.

Command ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null xxx "sudo su - -c \"docker ps  -a | grep container_e02_1531189225093_0003_01_000002\" root" failed after 0 retries