[YARN-7999] Docker launch fails when user private filecache directory is missing

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Labels: Docker
• Target Version/s: 3.1.0
• Assignee: Jason Darrell Lowe
• Reporter: Eric Yang
• Created: 02/Mar/18 23:33
• Updated: 02/May/18 22:34
• Resolved: 14/Mar/18 16:27

Description

Docker container is failing to launch in trunk. The root cause is:

[COMPINSTANCE sleeper-1 : container_1520032931921_0001_01_000020]: [2018-03-02 23:26:09.196]Exception from container-launch.
Container id: container_1520032931921_0001_01_000020
Exit code: 29
Exception message: image: hadoop/centos:latest is trusted in hadoop registry.
Could not determine real path of mount '/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache'
Could not determine real path of mount '/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache'
Invalid docker mount '/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache:/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache', realpath=/tmp/hadoop-yarn/nm-local-dir/usercache/hbase/filecache
Error constructing docker command, docker error code=12, error message='Invalid docker mount'
Shell output: main : command provided 4
main : run as user is hbase
main : requested yarn user is hbase
Creating script paths...
Creating local dirs...
[2018-03-02 23:26:09.240]Diagnostic message from attempt 0 : [2018-03-02 23:26:09.240]
[2018-03-02 23:26:09.240]Container exited with a non-zero exit code 29.
[2018-03-02 23:26:39.278]Could not find nmPrivate/application_1520032931921_0001/container_1520032931921_0001_01_000020//container_1520032931921_0001_01_000020.pid in any of the directories
[COMPONENT sleeper]: Failed 11 times, exceeded the limit - 10. Shutting down now...

The filecache cant not be mounted because it doesn't exist.

Issue Links

• is broken by: YARN-7815 Make the YARN mounts added to Docker containers more restrictive