[MESOS-4279] Docker executor truncates task's output when the task is killed.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.25.0, 0.26.0, 0.27.2, 0.28.1
• Fix Version/s: 1.0.0
• Component/s: containerization, docker
• Labels: docker mesosphere won't-backport
• Sprint: Mesosphere Sprint 35, Mesosphere Sprint 36
• Story Points: 5
• Assignee: Benjamin Mahler
• Reporter: Martin Bydzovsky
• Shepherd: Jie Yu
• Created: 04/Jan/16 15:57
• Updated: 26/Nov/18 12:22
• Resolved: 26/Nov/18 12:22

Description

I'm implementing a graceful restarts of our mesos-marathon-docker setup and I came to a following issue:

(it was already discussed on https://github.com/mesosphere/marathon/issues/2876 and guys form mesosphere got to a point that its probably a docker containerizer problem...)
To sum it up:

When i deploy simple python script to all mesos-slaves:

#!/usr/bin/python

from time import sleep
import signal
import sys
import datetime

def sigterm_handler(_signo, _stack_frame):
    print "got %i" % _signo
    print datetime.datetime.now().time()
    sys.stdout.flush()
    sleep(2)
    print datetime.datetime.now().time()
    print "ending"
    sys.stdout.flush()
    sys.exit(0)

signal.signal(signal.SIGTERM, sigterm_handler)
signal.signal(signal.SIGINT, sigterm_handler)

try:
    print "Hello"
    i = 0
    while True:
        i += 1
        print datetime.datetime.now().time()
        print "Iteration #%i" % i
        sys.stdout.flush()
        sleep(1)
finally:
    print "Goodbye"

and I run it through Marathon like

data = {
	args: ["/tmp/script.py"],
	instances: 1,
	cpus: 0.1,
	mem: 256,
	id: "marathon-test-api"
}

During the app restart I get expected result - the task receives sigterm and dies peacefully (during my script-specified 2 seconds period)

But when i wrap this python script in a docker:

FROM node:4.2

RUN mkdir /app
ADD . /app
WORKDIR /app
ENTRYPOINT []

and run appropriate application by Marathon:

data = {
	args: ["./script.py"],
	container: {
		type: "DOCKER",
		docker: {
			image: "bydga/marathon-test-api"
		},
		forcePullImage: yes
	},
	cpus: 0.1,
	mem: 256,
	instances: 1,
	id: "marathon-test-api"
}

The task during restart (issued from marathon) dies immediately without having a chance to do any cleanup.

Issue Links

• is related to: MESOS-5195 Docker executor: task logs lost on shutdown