[MESOS-7027] CommandExecutor ENV overwritten by Docker Image ENV in Unified Containerizer

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Labels: environment mesosphere
• Target Version/s: 1.2.0
• Flags: Important
• Sprint: Mesosphere Sprint 50, Mesosphere Sprint 51
• Story Points: 5
• Assignee: Joseph Wu
• Reporter: Kevin Klues
• Shepherd: Jie Yu
• Created: 28/Jan/17 02:52
• Updated: 06/May/17 00:42
• Resolved: 08/Feb/17 01:43

Description

Using the unified containerizer, if a docker image is provisioned and has environment variables set via the ENV directive, those environment variables will be inherited by the mesos-executor process and overwrite similarly named environment variables that otherwise would have been inherited from the agent.

This causes problems (for example) in DC/OS when trying to launch tasks based off the nvidia/cuda image. The nvidia/cuda image explicitly sets LD_LIBRARY_PATH to its own value so that the proper nvidia libraries will be available to whatever command is launched inside the container.

However, DC/OS relies on LD_LIBRARY_PATH to contain a path to /opt/mesosphere/lib so that all of the mesosphere libraries are available to the mesos binaries launched by the agent (mesos-containerizer, mesos-execute, etc.). This is necessary to make sure that any external dependencies they might have (e.g. libssl.so) can be resolved at runtime.
By overwriting the executor's environment with the Docker Image environment, LD_LIBRARY_PATH will not be set properly and mesos-execute will fail.

It seems to me, the Docker Image environment should only actually overwrite the environment of the user process (not its executor). However, this can get complicated, because the executor actually is the user process in the case of launching a custom executor.

We need to rethink how the environment is inherited/overwritten through all the various processes that get spawned while launching a container as well as how to make it work for tasks launched by arbitrary executors.

Issue Links

• is depended upon by: MESOS-7429 Allow isolators to inject task-specific environment variables.
• relates to: MESOS-4781 Executor env variables should not be leaked to the command task.
• relates to: MESOS-6648 MesosContainerizer launch helper should take ContainerLaunchInfo.