[MESOS-9462] Devices in a container are inaccessible due to `nodev` on `/var/run`.

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0
• Component/s: containerization
• Labels: containerizer regression
• Target Version/s: 1.8.0
• Sprint: Containerization R8 Sprint 35
• Story Points: 3
• Assignee: Andrei Budnik
• Reporter: Jie Yu
• Created: 07/Dec/18 19:04
• Updated: 11/Dec/18 08:07
• Resolved: 11/Dec/18 08:06

Description

A recent patch (commit ede8155d1d043137e15007c48da36ac5fa0b5124) changes the behavior of how standard device nodes (e.g., /dev/null, etc.) are setup. It uses bind mount (from host) now (instead of mknod).

The devices nodes are created under `/var/run/mesos/containers/<container_id>/devices`, and then bind mounted to the container root filesystem. This is problematic for those Linux distros that mount `/var/run` (or `/run`) as `nodev`. For instance, CentOS 7.4:

[jie@core-dev ~]$ cat /proc/self/mountinfo | grep "/run\ "                                                                                                                         
24 62 0:19 / /run rw,nosuid,nodev shared:23 - tmpfs tmpfs rw,seclabel,mode=755
[jie@core-dev ~]$ cat /etc/redhat-release 
CentOS Linux release 7.4.1708 (Core) 

As a result, the `/dev/null` devices in the container will inherit the `nodev` from `/run` on the host

629 625 0:121 /mesos/containers/49f1da14-d741-4030-994c-0d8ed5093b13/devices/null /dev/null rw,nosuid,nodev - tmpfs tmpfs rw,mode=755

This will cause "Permission Denied" error when a process in the container tries to open the device node.

You can try to reproduce this issue using Mesos Mini

docker run --rm --privileged -p 5050:5050 -p 5051:5051 -p 8080:8080 mesos/mesos-mini:master-2018-12-06

And the, go to Marathon UI (http://localhost:8080), and launch an app using the following config

{
  "id": "/test",
  "cmd": "dd if=/dev/zero of=file bs=1024 count=1 oflag=dsync",
  "cpus": 1,
  "mem": 128,
  "disk": 128,
  "instances": 1,
  "container": {
    "type": "MESOS",
    "docker": {
      "image": "ubuntu:18.04"
    }
  }
}

You'll see the task failed with "Permission Denied".

The task will run normally if you use `mesos/mesos-mini:master-2018-12-01`

Issue Links

• is caused by: MESOS-9319 Move root filesystem creation to the `filesystem/linux` isolator.