[MESOS-5307] Sandbox mounts should not be in the host mount namespace.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.28.2, 1.0.0
• Labels: mesosphere
• Epic Link: filesystem isolation
• Sprint: Mesosphere Sprint 34
• Story Points: 5
• Assignee: Jie Yu
• Reporter: Jie Yu
• Created: 29/Apr/16 01:18
• Updated: 04/Jun/16 18:04
• Resolved: 02/May/16 16:53

Description

Currently, if a container uses container image, we'll do a bind mount of its sandbox (<sandbox> -> <rootfs>/mnt/mesos/sandbox) in the host mount namespace.

However, doing the mounts in the host mount table is not ideal. That complicates both the cleanup path and the recovery path.

Instead, we can do the sandbox bind mount in the container's mount namespace so that cleanup and recovery will be greatly simplified. We can setup mount propagation properly so that persistent volumes mounted at <sandbox>/xxx can be propagated into the container.

Here is a simple proof of concept:

Console 1:

vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ ll .
total 12
drwxrwxr-x 3 vagrant vagrant 4096 Apr 25 16:05 ./
drwxrwxr-x 6 vagrant vagrant 4096 Apr 25 23:17 ../
drwxrwxr-x 5 vagrant vagrant 4096 Apr 25 23:17 slave/
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ ll slave/
total 20
drwxrwxr-x  5 vagrant vagrant 4096 Apr 25 23:17 ./
drwxrwxr-x  3 vagrant vagrant 4096 Apr 25 16:05 ../
drwxrwxr-x  6 vagrant vagrant 4096 Apr 26 21:06 directory/
drwxr-xr-x 12 vagrant vagrant 4096 Apr 25 23:20 rootfs/
drwxrwxr-x  2 vagrant vagrant 4096 Apr 25 16:09 volume/
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ sudo mount --bind slave/ slave/                                                                                                                                                                                                                            
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ sudo mount --make-shared slave/
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ cat /proc/self/mountinfo 
50 22 8:1 /home/vagrant/tmp/mesos/slave /home/vagrant/tmp/mesos/slave rw,relatime shared:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered

Console 2:

vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ cd slave/
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave$ sudo unshare -m /bin/bash
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# sudo mount --make-rslave .
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# cat /proc/self/mountinfo
124 63 8:1 /home/vagrant/tmp/mesos/slave /home/vagrant/tmp/mesos/slave rw,relatime master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# mount --rbind directory/ rootfs/mnt/mesos/sandbox/                                                                                                                                                                                        
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# mount --rbind rootfs/ rootfs/
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# mount -t proc proc rootfs/proc                                                                                                                                                                                                            
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# pivot_root rootfs rootfs/tmp/.rootfs                                                                                                                                                                                                      
root@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave# cd /
root@vagrant-ubuntu-trusty-64:/# cat /proc/self/mountinfo
126 61 8:1 /home/vagrant/tmp/mesos/slave/rootfs / rw,relatime master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
127 126 8:1 /home/vagrant/tmp/mesos/slave/directory /mnt/mesos/sandbox rw,relatime master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
128 126 0:3 / /proc rw,relatime - proc proc rw

Console 1:

agrant@vagrant-ubuntu-trusty-64:~/tmp/mesos$ cd slave/
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave$ sudo mount --bind volume/ directory/v1
vagrant@vagrant-ubuntu-trusty-64:~/tmp/mesos/slave$ cat /proc/self/mountinfo
50 22 8:1 /home/vagrant/tmp/mesos/slave /home/vagrant/tmp/mesos/slave rw,relatime shared:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
129 50 8:1 /home/vagrant/tmp/mesos/slave/volume /home/vagrant/tmp/mesos/slave/directory/v1 rw,relatime shared:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered

Console 2:

root@vagrant-ubuntu-trusty-64:/# cat /proc/self/mountinfo
126 61 8:1 /home/vagrant/tmp/mesos/slave/rootfs / rw,relatime master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
127 126 8:1 /home/vagrant/tmp/mesos/slave/directory /mnt/mesos/sandbox rw,relatime master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered
128 126 0:3 / /proc rw,relatime - proc proc rw
132 127 8:1 /home/vagrant/tmp/mesos/slave/volume /mnt/mesos/sandbox/v1 rw,relatime shared:4 master:1 - ext4 /dev/disk/by-uuid/baf292e5-0bb6-4e58-8a71-5b912e0f09b6 rw,data=ordered

Issue Links

• is related to: MESOS-5306 Mounting /etc/xxx files in CNI for command tasks has race condition.