Details
-
Story
-
Status: Resolved
-
Minor
-
Resolution: Won't Fix
-
None
-
None
-
Ubuntu 11.04, Ubuntu 11.10
Description
Earlier in the year I implemented a virtual machine isolation module. This module uses lib-virt to launch and manage virtual machine containers. The code is still rough and have done basic testing with the Spark example.
This code works with the KVM (http://www.linux-kvm.org/page/Main_Page) virtual machine manager. I've placed the relevant code in a branch called mesos-vm, for now located at https://github.com/charlescearl/VirtualMesos. The code is based upon the mesos lxc isolation module that is located in src/slave/lxc_isolation_module.cpp/.hpp. My code based on the mesos master branch dated Wed Nov 23 12:02:07 2011 -0800, commit 059aabb2ec5bd7b20ed08ab9c439531a352ba3ec. I'll generate a patch soon for this. Suggestions appreciated on whether this is the appropriate branch/commit to patch against.
Most of the implementation is contained in vm_isolation_module.cpp and vm_isolation_module.hpp and there are some minor additions in launcher to handle setup of the environment for the virtual machine. I use the libvirt (http://libvirt.org/) library, to manage the virtual machine container in which the jobs are executed.
Dependencies
The code has been tested on Ubuntu 11.04 and 11.10 and depends on libpython2.6 and libvirt0
Configuration of the virtual machine container
The virtual machine invocation depends upon a few configuration assumptions:
1. ssh public keys installed on the container. I assume that the container is setup to allow password-less secure access.
2. Directory structure on the container matches the servant machine. For example, in invoking a spark executor, assume that the paths match the setup on the container host.
Running it
In the $MESOS_HOME/conf/mesos.conf file add the line
isolation=vm
to use the virtual machine isolation.
The Mesos slave is invoked with the isolation parameter set to vm. For example
sudo bin/mesos-slave -m mesos://master@mesos-host:5050 -w 9839 --isolation=vm
Rough description of how it works
The `vm_isolation_module` class forks a process that in turn launches a virtual machine. A routine located in bin called find_addr.pl is responsible for figuring out the IP address of the launched virtual machine. This is probably not portable since it is explicitly looking for entry in the virbr0 network.
A script vmLauncherTemplate.sh located in bin assists the the vmLauncher method to setup the environment for launching tasks inside of the virtual machine. The vmLauncher method uses vmLauncherTemplate.sh to create a tasks specific shell vmLauncherTemplate-<task_id>.sh, which is copied to the running guest and used to run the executor inside the VM. This communicates with the slave on the host.
Comments and suggestions on improvements and next directions are appreciated!
Attachments
Issue Links
- is superceded by
-
MESOS-2717 Qemu/KVM containerizer
-
- Accepted
-