[AURORA-1902] Docker containers with not newest OS fails to run

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.17.0
• Component/s: Docker, Executor

Environment

Ubuntu: 16.04
Mesos: 1.1.0
Aurora: 0.17.0
Dockerengine: 1.13.1

• Assignee: Unassigned
• Reporter: Mikhail Lesyk
• Created: 08/Mar/17 20:01
• Updated: 08/Mar/17 21:24

Description

When trying to launch Docker containers, got an error:

I0308 21:47:56.695737  3888 fetcher.cpp:498] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7","items":[{"action":"BYPASS_CACHE","uri":{"executable":true,"extract":true,"value":"\/usr\/share\/aurora\/bin\/thermos_executor.pex"}}],"sandbox_directory":"\/var\/lib\/mesos\/slaves\/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7\/frameworks\/47934424-623f-4fcb-9326-bf668149fc77-0000\/executors\/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261\/runs\/788d2f72-a6eb-4f3e-999c-17158e473661"}
I0308 21:47:56.701079  3888 fetcher.cpp:409] Fetching URI '/usr/share/aurora/bin/thermos_executor.pex'
I0308 21:47:56.701162  3888 fetcher.cpp:250] Fetching directly into the sandbox directory
I0308 21:47:56.701225  3888 fetcher.cpp:187] Fetching URI '/usr/share/aurora/bin/thermos_executor.pex'
I0308 21:47:56.701282  3888 fetcher.cpp:167] Copying resource with command:cp '/usr/share/aurora/bin/thermos_executor.pex' '/var/lib/mesos/slaves/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7/frameworks/47934424-623f-4fcb-9326-bf668149fc77-0000/executors/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261/runs/788d2f72-a6eb-4f3e-999c-17158e473661/thermos_executor.pex'
I0308 21:47:56.730024  3888 fetcher.cpp:547] Fetched '/usr/share/aurora/bin/thermos_executor.pex' to '/var/lib/mesos/slaves/7cbc133f-24ac-4937-aa28-09e8c81b647b-S7/frameworks/47934424-623f-4fcb-9326-bf668149fc77-0000/executors/thermos-root-prod-test-0-a2c19f58-aa6c-45d8-a47f-8cf57dc0c261/runs/788d2f72-a6eb-4f3e-999c-17158e473661/thermos_executor.pex'
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Traceback (most recent call last):
  File "apache/aurora/executor/bin/thermos_executor_main.py", line 45, in <module>
    from mesos.executor import MesosExecutorDriver
  File "/root/.pex/install/mesos.executor-1.1.0-py2.7-linux-x86_64.egg.47fa022c99c11c7faddf379cbfc46a25c5f215be/mesos.executor-1.1.0-py2.7-linux-x86_64.egg/mesos/executor/__init__.py", line 17, in <module>
    from ._executor import MesosExecutorDriverImpl as MesosExecutorDriver
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /root/.pex/install/mesos.executor-1.1.0-py2.7-linux-x86_64.egg.47fa022c99c11c7faddf379cbfc46a25c5f215be/mesos.executor-1.1.0-py2.7-linux-x86_64.egg/mesos/executor/_executor.so)

twitter.common.app debug: Initializing: twitter.common.log (Logging subsystem.)
Writing log files to disk in /mnt/mesos/sandbox
thermos_executor.pex: error: Could not load MesosExecutorDriver!
twitter.common.app debug: main sys.exited
twitter.common.app debug: Shutting application down.
twitter.common.app debug: Running exit function for twitter.common.log (Logging subsystem.)
twitter.common.app debug: Finishing up module teardown.
twitter.common.app debug:   Active thread: <_MainThread(MainThread, started 140218447816512)>
twitter.common.app debug: Exiting cleanly.

Tested affected systems(with absent of GLIBCXX_3.4.20, GLIBCXX_3.4.21):
Debian 8
Ubuntu 14.04

How to reproduce:
1) Prepare Docker image with Python 2.7. Example of Dockerfile:

FROM ubuntu:14.04

RUN apt-get -y update && apt-get -y install python2.7

2) build and push image to some repo, example:

docker build -t mlesyk/ubuntu:14.04 . && docker push mlesyk/ubuntu:14.04

3) create some job with Docker container with any command to run, for example,

sleep 60

and appropriate container parameter, for example:

    container = Docker(image='mlesyk/ubuntu:14.04')

4) Run this job in Aurora and observe error from beginning of ticket