[MESOS-9536] Nested container launched with non-root user may not be able to write to its sandbox via the environment variable `MESOS_SANDBOX`

• Type: Bug
• Status: Accepted
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.6.0, 1.6.1, 1.7.0, 1.8.0
• Component/s: containerization
• Sprint: Containerization RI10 Spr 38, Containerization RI12 Sp 43, Containerization: RI-13 Sp 44, Containerization: RI13 Sp 45, Containerization: RI-18 54
• Story Points: 3
• Assignee: Qian Zhang
• Reporter: Qian Zhang
• Shepherd: Gilbert Song
• Created: 25/Jan/19 08:12
• Updated: 11/Sep/19 22:36

Description

Launch a nested container to write to its sandbox via the env var `MESOS_SANDBOX`. The nested container is launched with a non-root user (e.g., `nobody`) and its parent container (i.e., the default executor) is launched with root since `mesos-execute` is executed with `sudo` in the example below.

$ sudo src/mesos-execute --master=<master-IP>:5050 --task_group=file:///tmp/task_group.json
$ cat /tmp/task_group.json
{
  "tasks":[
    {
      "name" : "test",
      "task_id" : {"value" : "test"},
      "agent_id": {"value" : ""},
      "resources": [
        {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
        {"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
      ],
      "command": {
        "user": "nobody",
        "value": "echo data > $MESOS_SANDBOX/file"
      }
    }
  ]
}

The nested container will fail.

I0125 16:04:03.610659 10064 scheduler.cpp:189] Version: 1.8.0
I0125 16:04:03.641856 10066 scheduler.cpp:355] Using default 'basic' HTTP authenticatee
I0125 16:04:03.643841 10063 scheduler.cpp:538] New master detected at master@192.168.56.5:5050
Subscribed with ID 1ae64562-dbf9-4b24-af88-1cbcdc2ae71d-0002
Submitted task group with tasks [ test ] to agent '12866186-dc2b-48a9-88ad-f9d951cf8c7f-S0'
Received status update TASK_STARTING for task 'test'
  source: SOURCE_EXECUTOR
Received status update TASK_RUNNING for task 'test'
  source: SOURCE_EXECUTOR
Received status update TASK_FAILED for task 'test'
  message: 'Command exited with status 2'
  source: SOURCE_EXECUTOR

In the stderr of the nested container, we can see it has no permission to do the write.

$ sudo cat /opt/mesos/slaves/12866186-dc2b-48a9-88ad-f9d951cf8c7f-S0/frameworks/1ae64562-dbf9-4b24-af88-1cbcdc2ae71d-0002/executors/default-executor/runs/c7173fd8-9c01-49f5-a092-bdad78609260/containers/bf8f6ac8-2f8a-4300-9fe6-a830f602f654/stderr 
Marked '/' as rslave
sh: 1: cannot create /opt/mesos/slaves/12866186-dc2b-48a9-88ad-f9d951cf8c7f-S0/frameworks/1ae64562-dbf9-4b24-af88-1cbcdc2ae71d-0002/executors/default-executor/runs/c7173fd8-9c01-49f5-a092-bdad78609260/containers/bf8f6ac8-2f8a-4300-9fe6-a830f602f654/file: Permission denied