[AURORA-1781] Sandbox taskfs setup fails (groupadd error)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.16.0
• Component/s: Docker, Executor
• Assignee: Unassigned
• Reporter: Justin Venus
• Created: 24/Sep/16 19:56
• Updated: 21/Sep/17 22:09

Description

I hit what smells like a permission issue w/ `/etc/group` when trying to use a docker-image (unified containerizer setup) with mesos-1.0.0. and aurora-0.16.0-rc2. I cannot reproduce issue w/ mesos-0.28.2 and aurora-015.0.

Failed to initialize sandbox: Failed to create group in sandbox for task image: Command '['groupadd', '-R', '/var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-0000/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs', '-g', '99', 'nobody']' returned non-zero exit status 10

[root@mesos-master01of2 taskfs]# pwd
/var/lib/mesos/slaves/5d28d0cc-2793-4471-82d5-e67276c53f70-S2/frameworks/20160221-001235-3801519626-5050-1-0000/executors/thermos-nobody-prod-jenkins-0-47cc7824-565b-4265-9ab4-9ba3f364ebed/runs/a3f78288-4865-4166-8685-1ad941562f2f/taskfs
[root@mesos-master01of2 taskfs]# groupadd -R $PWD -g 99 nobody
groupadd: cannot lock /etc/group; try again later.

Maybe related to AURORA-1761

I'm running CoreOS with the mesos-agent (and thermos) inside docker. Here is the gist of how it's started.

/usr/bin/sh -c "exec /usr/bin/docker run \
    --name=mesos_slave \
    --net=host \
    --pid=host \
    --privileged \
    -v /sys:/sys \
    -v /usr/bin/docker:/usr/bin/docker:ro \
    -v /var/lib/docker:/var/lib/docker \
    -v /var/run/docker.sock:/root/docker.sock \
    -v /run/systemd/system:/run/systemd/system \
    -v /lib64/libdevmapper.so.1.02:/lib/libdevmapper.so.1.02:ro \
    -v /sys/fs/cgroup:/sys/fs/cgroup \
    -v /var/lib/mesos:/var/lib/mesos \
    -e MESOS_CONTAINERIZERS=docker,mesos \
    -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins \
    -e MESOS_WORK_DIR=/var/lib/mesos \
    -e MESOS_LOGGING_LEVEL=INFO \
    -e AMAZON_REGION=us-office-2 \
    -e AVAILABILITY_ZONE=us-office-2b \
    -e MESOS_ATTRIBUTES=\"platform:linux;host:$(hostname);rack:us-office-2b\" \
    -e MESOS_CLUSTER=ZeroZero \
    -e MESOS_DOCKER_SOCKET=/root/docker.sock \
    -e MESOS_MASTER=zk://10.150.150.224:2181,10.150.150.225:2181,10.150.150.226:2181/mesos \
    -e MESOS_LOG_DIR=/var/log/mesos \
    -e MESOS_ISOLATION=\"filesystem/linux,cgroups/cpu,cgroups/mem,docker/runtime\" \
    -e MESOS_IMAGE_PROVIDERS=docker \
    -e MESOS_IMAGE_PROVISIONER_BACKEND=copy \
    -e MESOS_DOCKER_REGISTRY=http://docker-registry:31000 \
    -e MESOS_DOCKER_STORE_DIR=/var/lib/mesos/docker \
    --entrypoint=/usr/sbin/mesos-slave \
    docker-registry.thebrighttag.com:31000/mesos:latest \
        --no-systemd_enable_support \
    || rm -f /var/lib/mesos/meta/slaves/latest"