Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3136

COMMAND health checks with Marathon 0.10.0 are broken

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1, 0.24.1, 0.25.0
    • Component/s: None
    • Labels:
      None

      Description

      When deploying Mesos 0.23rc4 with latest Marathon 0.10.0 RC3 command health check stop working. Rolling back to Mesos 0.22.1 fixes the problem.

      Containerizer is Docker.
      All packages are from official Mesosphere Ubuntu 14.04 sources.

      The issue must be analyzed further.

      1. MESOS-3136_0_24_0.patch
        20 kB
        haosdent
      2. MESOS-3136_0_23_0.patch
        20 kB
        haosdent

        Issue Links

          Activity

          Hide
          ekesken Erhan Kesken added a comment -

          I cloned this issue as adam suggested: MESOS-3599, thank you.

          Show
          ekesken Erhan Kesken added a comment - I cloned this issue as adam suggested: MESOS-3599 , thank you.
          Hide
          ekesken Erhan Kesken added a comment -

          sorry for my lack of attention, I had used parameter as --launcher_dir as it supposed to be, I'd written wrong versions at my comment here.

          Show
          ekesken Erhan Kesken added a comment - sorry for my lack of attention, I had used parameter as --launcher_dir as it supposed to be, I'd written wrong versions at my comment here.
          Hide
          ekesken Erhan Kesken added a comment -

          as you stated, I think both option is required, at least at mesos-slave context times we had a chance to apply a workaround to hold both options in our hands, with an ugly line like this:

          docker exec $(for i in $(docker ps -q --no-trunc); do docker inspect $i | grep -sq MESOS_TASK_ID=${MESOS_TASK_ID:?} && echo $i; done) ls /
          

          but now in docker context there is no workaround for getting other option. so I see issue as a MUST.

          Show
          ekesken Erhan Kesken added a comment - as you stated, I think both option is required, at least at mesos-slave context times we had a chance to apply a workaround to hold both options in our hands, with an ugly line like this: docker exec $(for i in $(docker ps -q --no-trunc); do docker inspect $i | grep -sq MESOS_TASK_ID=${MESOS_TASK_ID:?} && echo $i; done) ls / but now in docker context there is no workaround for getting other option. so I see issue as a MUST.
          Hide
          haosdent@gmail.com haosdent added a comment - - edited

          I remember should be

          --launcher_dir=

          not

          --launch-dir

          . you missing

          er

          and use wrong

          -

          . And if you think running health check outside docker container is necessary, please feel free to open a new ticket as adam said. If that issue is acceptted, I would submit a patch for that as soon as possible.

          Show
          haosdent@gmail.com haosdent added a comment - - edited I remember should be --launcher_dir= not --launch-dir . you missing er and use wrong - . And if you think running health check outside docker container is necessary, please feel free to open a new ticket as adam said. If that issue is acceptted, I would submit a patch for that as soon as possible.
          Hide
          ekesken Erhan Kesken added a comment -

          both mesos-docker-executor and mesos-health-check binaries are under /usr/libexec/mesos folder.

          before passing --launch-dir to mesos-slave, mesos-docker-executor was failing, launch_dir fixed that problem but at next step health_check failed.

          when I checked code at https://github.com/apache/mesos/blob/5058fac1083dc91bca54d33c26c810c17ad95dd1/src/docker/executor.cpp#L573, I concluded that --launch-dir parameter does not set MESOS_LAUNCHER_DIR environment variable.

          I removed --launch-dir parameter and put MESOS_LAUNCHER_DIR enviroment setting to mesos-slave process by editing /etc/default/mesos-slave file. I confirmed new enviroment setting is active by checking /proc/XXXX/environ file. mesos-docker-executor did not fail like I passed --launch-dir parameter, but mesos-health-check failed again.

          I checked mesos-docker-executor process environment from proc, but MESOS_LAUNCHER_DIR was not there. finally I solved problem by putting environment variable to my marathon config after understanding health checks are running inside docker.

          Show
          ekesken Erhan Kesken added a comment - both mesos-docker-executor and mesos-health-check binaries are under /usr/libexec/mesos folder. before passing --launch-dir to mesos-slave, mesos-docker-executor was failing, launch_dir fixed that problem but at next step health_check failed. when I checked code at https://github.com/apache/mesos/blob/5058fac1083dc91bca54d33c26c810c17ad95dd1/src/docker/executor.cpp#L573 , I concluded that --launch-dir parameter does not set MESOS_LAUNCHER_DIR environment variable. I removed --launch-dir parameter and put MESOS_LAUNCHER_DIR enviroment setting to mesos-slave process by editing /etc/default/mesos-slave file. I confirmed new enviroment setting is active by checking /proc/XXXX/environ file. mesos-docker-executor did not fail like I passed --launch-dir parameter, but mesos-health-check failed again. I checked mesos-docker-executor process environment from proc, but MESOS_LAUNCHER_DIR was not there. finally I solved problem by putting environment variable to my marathon config after understanding health checks are running inside docker.
          Hide
          adam-mesos Adam B added a comment -

          I can understand the desire to run in the docker, so that it has access to all the environment/deps inside the container; or wanting to run it outside the docker so you could query the docker engine itself. Perhaps only outside the container is necessary, because the healthcheck command could specify to run `docker exec whatever` if it wants to run something inside the container.

          If we decide that this is an issue to pursue, let's please open a new JIRA (clone this one, or otherwise link it), so we can track the new issue separately from the one we've already shipped a "fix" for. Thanks!

          Show
          adam-mesos Adam B added a comment - I can understand the desire to run in the docker, so that it has access to all the environment/deps inside the container; or wanting to run it outside the docker so you could query the docker engine itself. Perhaps only outside the container is necessary, because the healthcheck command could specify to run `docker exec whatever` if it wants to run something inside the container. If we decide that this is an issue to pursue, let's please open a new JIRA (clone this one, or otherwise link it), so we can track the new issue separately from the one we've already shipped a "fix" for. Thanks!
          Hide
          haosdent@gmail.com haosdent added a comment -

          You need set MESOS_LAUNCHER_DIR or make sure mesos-docker-executor and mesos-health-check under same folder.

          Show
          haosdent@gmail.com haosdent added a comment - You need set MESOS_LAUNCHER_DIR or make sure mesos-docker-executor and mesos-health-check under same folder.
          Hide
          haosdent@gmail.com haosdent added a comment -

          A bit confuse, should the health check running outside docker or in docker? In our before discussion, we think should put the health check inside container, so we use "docker exec".

          Show
          haosdent@gmail.com haosdent added a comment - A bit confuse, should the health check running outside docker or in docker? In our before discussion, we think should put the health check inside container, so we use "docker exec".
          Hide
          ekesken Erhan Kesken added a comment -

          I tested mesos 0.24.1 with marathon 0.11.0, I'm using puppet module https://github.com/deric/puppet-mesos to install mesos package, I needed to add --launch-dir=/usr/libexec/mesos parameter to mesos-slave process, but that was not enough I also needed to put "MESOS_LAUNCHER_DIR": "/usr/libexec/mesos" line into env dict of my marathon config as well, otherwise launchhealthcheck command can not find place of mesos-health-check command. Is there a more proper solution for this problem?

          Show
          ekesken Erhan Kesken added a comment - I tested mesos 0.24.1 with marathon 0.11.0, I'm using puppet module https://github.com/deric/puppet-mesos to install mesos package, I needed to add --launch-dir=/usr/libexec/mesos parameter to mesos-slave process, but that was not enough I also needed to put "MESOS_LAUNCHER_DIR": "/usr/libexec/mesos" line into env dict of my marathon config as well, otherwise launchhealthcheck command can not find place of mesos-health-check command. Is there a more proper solution for this problem?
          Hide
          ekesken Erhan Kesken added a comment -

          Issue is marked as resolved, but from line https://github.com/apache/mesos/blob/5058fac1083dc91bca54d33c26c810c17ad95dd1/src/docker/executor.cpp#L326 I understand that health check commands are executed in docker context now, but old behaviour was running at slave context.

          As you can see at this thread https://groups.google.com/forum/#!topic/marathon-framework/S_k6k3ydQ-o , I believe many people currently have health check commands based on 'docker

          {exec,logs}

          ' commands, including me.

          even basic example at marathon docs https://mesosphere.github.io/marathon/docs/health-checks.html

          {
            "protocol": "COMMAND",
            "command": { "value": "curl -f -X GET http://$HOST:$PORT0/health" },
            "gracePeriodSeconds": 300,
            "intervalSeconds": 60,
            "timeoutSeconds": 20,
            "maxConsecutiveFailures": 3
          }
          

          won't work, because you don't have curl package in your docker image most probably.

          I think there should be a flag to get back old behaviour which was using slave context to execute health check commands to avoid people adding extra packages to their docker images and rewrite their health check commands.

          Show
          ekesken Erhan Kesken added a comment - Issue is marked as resolved, but from line https://github.com/apache/mesos/blob/5058fac1083dc91bca54d33c26c810c17ad95dd1/src/docker/executor.cpp#L326 I understand that health check commands are executed in docker context now, but old behaviour was running at slave context. As you can see at this thread https://groups.google.com/forum/#!topic/marathon-framework/S_k6k3ydQ-o , I believe many people currently have health check commands based on 'docker {exec,logs} ' commands, including me. even basic example at marathon docs https://mesosphere.github.io/marathon/docs/health-checks.html { "protocol": "COMMAND", "command": { "value": "curl -f -X GET http://$HOST:$PORT0/health" }, "gracePeriodSeconds": 300, "intervalSeconds": 60, "timeoutSeconds": 20, "maxConsecutiveFailures": 3 } won't work, because you don't have curl package in your docker image most probably. I think there should be a flag to get back old behaviour which was using slave context to execute health check commands to avoid people adding extra packages to their docker images and rewrite their health check commands.
          Hide
          haosdent@gmail.com haosdent added a comment - - edited

          The backport patches for 0.23.0 and 0.24.0-rc1 are in attached files. Test with

          sudo GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter="HealthCheckTest*" --verbose
          
          Show
          haosdent@gmail.com haosdent added a comment - - edited The backport patches for 0.23.0 and 0.24.0-rc1 are in attached files. Test with sudo GLOG_v=1 ./bin/mesos-tests.sh --gtest_filter= "HealthCheckTest*" --verbose
          Hide
          tnachen Timothy Chen added a comment -

          commit 5d6a06082ce396582add2e3e8497e1f4cdf91ec1
          Author: haosdent huang <haosdent@gmail.com>
          Date: Wed Sep 16 17:54:26 2015 +0000

          Added health check support for docker command tasks.

          Review: https://reviews.apache.org/r/37505

          Show
          tnachen Timothy Chen added a comment - commit 5d6a06082ce396582add2e3e8497e1f4cdf91ec1 Author: haosdent huang <haosdent@gmail.com> Date: Wed Sep 16 17:54:26 2015 +0000 Added health check support for docker command tasks. Review: https://reviews.apache.org/r/37505
          Hide
          adam-mesos Adam B added a comment -

          Greg Mann is doing some (internal) backport testing for 0.21.2,0.22.2,0.23.1,0.24.1 for the docker versioning patches from MESOS-2986 (one of which you haosdent wrote). Although this patch is likely unrelated to those others, if we can land it soon, it may be critical enough to include in at least one of those patch releases. Let's bring it up on the release proposal email thread: http://search-hadoop.com/m/0Vlr6PBeaOUhF241

          Show
          adam-mesos Adam B added a comment - Greg Mann is doing some (internal) backport testing for 0.21.2,0.22.2,0.23.1,0.24.1 for the docker versioning patches from MESOS-2986 (one of which you haosdent wrote). Although this patch is likely unrelated to those others, if we can land it soon, it may be critical enough to include in at least one of those patch releases. Let's bring it up on the release proposal email thread: http://search-hadoop.com/m/0Vlr6PBeaOUhF241
          Hide
          haosdent@gmail.com haosdent added a comment -

          Do you have a link to Greg Mann's work? I could not find it.

          Show
          haosdent@gmail.com haosdent added a comment - Do you have a link to Greg Mann 's work? I could not find it.
          Hide
          marco-mesos Marco Massenzio added a comment -

          I'm wondering how does this relate to the work Greg Mann is already doing around Docker versioning?
          As for me, I'd love it if we could fix the issue with Marathon, obviously!

          We could probably piggy-back this CL to that one too?

          • Timothy Chen could you please do a final review round, and see if you're happy with the latest patch?
          • Adam B I'll leave it up to you to coordinate and decide whether to pull this in in 0.24.1 and 0.23.x.

          Thanks!

          Show
          marco-mesos Marco Massenzio added a comment - I'm wondering how does this relate to the work Greg Mann is already doing around Docker versioning? As for me, I'd love it if we could fix the issue with Marathon, obviously! We could probably piggy-back this CL to that one too? Timothy Chen could you please do a final review round, and see if you're happy with the latest patch? Adam B I'll leave it up to you to coordinate and decide whether to pull this in in 0.24.1 and 0.23.x . Thanks!
          Hide
          haosdent@gmail.com haosdent added a comment -

          Marco MassenzioAdam BTimothy Chen If we could backport this to 0.23, I could attach a backport patch to 0.23.

          Show
          haosdent@gmail.com haosdent added a comment - Marco Massenzio Adam B Timothy Chen If we could backport this to 0.23, I could attach a backport patch to 0.23.
          Hide
          meatmanek Evan Krall added a comment -

          We also have Marathon jobs that are relying on command healthchecks on mesos 0.22.1. I hope that a fix for this issue gets backported to 0.23.x and 0.24.x.

          Show
          meatmanek Evan Krall added a comment - We also have Marathon jobs that are relying on command healthchecks on mesos 0.22.1. I hope that a fix for this issue gets backported to 0.23.x and 0.24.x.
          Hide
          vinodkone Vinod Kone added a comment -

          I'm afraid not considering this is not new in 24.0 and the vote is already out.

          Show
          vinodkone Vinod Kone added a comment - I'm afraid not considering this is not new in 24.0 and the vote is already out.
          Hide
          sybrandy Casey Sybrandy added a comment -

          I'm seeing this issue as well. Is there a chance it will be part of the 0.24 release?

          Thanks.

          Show
          sybrandy Casey Sybrandy added a comment - I'm seeing this issue as well. Is there a chance it will be part of the 0.24 release? Thanks.
          Hide
          marco-mesos Marco Massenzio added a comment -

          Vinod Kone does this need to be fixed before 0.24 is out?

          Show
          marco-mesos Marco Massenzio added a comment - Vinod Kone does this need to be fixed before 0.24 is out?
          Hide
          marco-mesos Marco Massenzio added a comment -

          Just noticed this entirely randomly...

          I would strongly suggest to avoid skipping a version between 0.22 / 0.24 as the Leader Election would be terminally broken: we transitioned to JSON in ZK for MasterInfo and while the 0.22 --> 0.23 --> 0.24 chain all works just fine, skipping 0.23 would create no end of grief.

          (I'm almost sure other stuff around HTTP API would break, but not sure there).

          My 2c

          Show
          marco-mesos Marco Massenzio added a comment - Just noticed this entirely randomly... I would strongly suggest to avoid skipping a version between 0.22 / 0.24 as the Leader Election would be terminally broken: we transitioned to JSON in ZK for MasterInfo and while the 0.22 --> 0.23 --> 0.24 chain all works just fine, skipping 0.23 would create no end of grief. (I'm almost sure other stuff around HTTP API would break, but not sure there). My 2c
          Hide
          haosdent@gmail.com haosdent added a comment -

          Patch: https://reviews.apache.org/r/37505/

          Timothy ChenAdam B Could you help review this patch? I am not sure my way is best way to add health check support to docker executor or not.

          Show
          haosdent@gmail.com haosdent added a comment - Patch: https://reviews.apache.org/r/37505/ Timothy Chen Adam B Could you help review this patch? I am not sure my way is best way to add health check support to docker executor or not.
          Hide
          adam-mesos Adam B added a comment -

          We generally don't support skipping a version in a live rolling upgrade, since we use each version as an opportunity to deprecate then remove old flags/endpoints/APIs. But so as long as nothing from 0.22 has been completely removed in 0.24, a skip upgrade could theoretically work.

          Show
          adam-mesos Adam B added a comment - We generally don't support skipping a version in a live rolling upgrade, since we use each version as an opportunity to deprecate then remove old flags/endpoints/APIs. But so as long as nothing from 0.22 has been completely removed in 0.24, a skip upgrade could theoretically work.
          Hide
          haosdent@gmail.com haosdent added a comment -

          If 0.22 users could upgrade to 0.24 directly?

          Show
          haosdent@gmail.com haosdent added a comment - If 0.22 users could upgrade to 0.24 directly?
          Hide
          adam-mesos Adam B added a comment -

          No. Mesos 0.23.0 was released the day before this issue was filed. We could package the fix into a 0.23.1 if it's critical.
          However, now that we're moving towards a monthly release cycle, we plan to cut a 0.24.0 release candidate (rc1) soon (next week?), so it might make more sense to just focus on getting somebody to fix this before 0.24.0-rc1 comes out.

          (Adding some metadata in hopes that this'll get more attention.)

          Show
          adam-mesos Adam B added a comment - No. Mesos 0.23.0 was released the day before this issue was filed. We could package the fix into a 0.23.1 if it's critical. However, now that we're moving towards a monthly release cycle, we plan to cut a 0.24.0 release candidate (rc1) soon (next week?), so it might make more sense to just focus on getting somebody to fix this before 0.24.0-rc1 comes out. (Adding some metadata in hopes that this'll get more attention.)
          Hide
          sttts Dr. Stefan Schimanski added a comment -

          Is there a chance to get a fix into 0.23.0?

          Show
          sttts Dr. Stefan Schimanski added a comment - Is there a chance to get a fix into 0.23.0?
          Hide
          haosdent@gmail.com haosdent added a comment -

          Currently only CommandExecutorProcess have launchHealthCheck. Maybe we could create a common class named ExecutorProcess and let CommandExecutorProcess/DockerExecutorProcess inherit from it? So that we could call launchHealthCheck in DockerExecutorProcess.

          Show
          haosdent@gmail.com haosdent added a comment - Currently only CommandExecutorProcess have launchHealthCheck. Maybe we could create a common class named ExecutorProcess and let CommandExecutorProcess/DockerExecutorProcess inherit from it? So that we could call launchHealthCheck in DockerExecutorProcess.
          Hide
          sttts Dr. Stefan Schimanski added a comment -

          Timothy Chen I think you hit the nail on the head. The issue I have encountered is exactly that: the new docker executor does not support the task health checks yet, the old one in 0.22 did.

          Now the actual problem is that Marathon supports those COMMAND health checks for a long time and people use it a lot. Hence, this looks like a regression in 0.23+ for the Marathon users.

          Would it be feasible to fix that by taking (or refactoring) the launcher/executor.cpp code for the docker executor? Or in fact, it would be even more awesome (although also kind of incompatible with the old behavior) if the new docker executor could execute the health checks inside the corresponding container via "docker exec". Kubernetes is following this route as well.

          Show
          sttts Dr. Stefan Schimanski added a comment - Timothy Chen I think you hit the nail on the head. The issue I have encountered is exactly that: the new docker executor does not support the task health checks yet, the old one in 0.22 did. Now the actual problem is that Marathon supports those COMMAND health checks for a long time and people use it a lot. Hence, this looks like a regression in 0.23+ for the Marathon users. Would it be feasible to fix that by taking (or refactoring) the launcher/executor.cpp code for the docker executor? Or in fact, it would be even more awesome (although also kind of incompatible with the old behavior) if the new docker executor could execute the health checks inside the corresponding container via "docker exec". Kubernetes is following this route as well.
          Hide
          sttts Dr. Stefan Schimanski added a comment -

          I have a second cluster now with the same behavior. Good news: I can leave it on 0.23 and debug tomorrow.

          Show
          sttts Dr. Stefan Schimanski added a comment - I have a second cluster now with the same behavior. Good news: I can leave it on 0.23 and debug tomorrow.
          Hide
          sttts Dr. Stefan Schimanski added a comment -

          HTTP and TCP health checks are done by Marathon itself. Command health checks are done via Mesos.

          Show
          sttts Dr. Stefan Schimanski added a comment - HTTP and TCP health checks are done by Marathon itself. Command health checks are done via Mesos.
          Hide
          tnachen Timothy Chen added a comment -

          Is COMMAND health check in Marathon using Mesos's health check function, or Marathon is doing healthchecking itself somehow?
          It's actually not a supported scenario yet for supporting Mesos's healthcheck with DockerContainerizer, since it doesn't really perform the command in the docker container. We should create a ticket and work on that in the future.

          Show
          tnachen Timothy Chen added a comment - Is COMMAND health check in Marathon using Mesos's health check function, or Marathon is doing healthchecking itself somehow? It's actually not a supported scenario yet for supporting Mesos's healthcheck with DockerContainerizer, since it doesn't really perform the command in the docker container. We should create a ticket and work on that in the future.
          Hide
          sttts Dr. Stefan Schimanski added a comment - - edited

          The app was part of a bigger Marathon application groups. It essentially looks like this:

          Note that for the health check to work the redis-tools package (Ubuntu) with the redis-cli must be installed on the slave.

          {
          	"id": "redis",
          	"container": {
          		"type": "DOCKER",
          		"docker": {
          			"image": "redis:2.8",
          			"network": "HOST"
          		}
          	},
          	"cmd": "touch /tmp/redis.conf; redis-server /tmp/redis.conf --dir /tmp --loglevel verbose",
          	"cpus": 0.2,
          	"mem": 256.0,
          	"instances": 1,
          	"maxLaunchDelaySeconds": 5,
          	"healthChecks": [{
          		"protocol": "COMMAND",
          		"command": { "value": "env; redis-cli -h $HOST -n 1 info | egrep 'role:master|master_link_status:up'" },
          		"gracePeriodSeconds": 20,
          		"intervalSeconds": 20,
          		"portIndex": 0,
          		"timeoutSeconds": 10,
          		"maxConsecutiveFailures": 3
          	}]
          }
          
          Show
          sttts Dr. Stefan Schimanski added a comment - - edited The app was part of a bigger Marathon application groups. It essentially looks like this: Note that for the health check to work the redis-tools package (Ubuntu) with the redis-cli must be installed on the slave. { "id" : "redis" , "container" : { "type" : "DOCKER" , "docker" : { "image" : "redis:2.8" , "network" : "HOST" } }, "cmd" : "touch /tmp/redis.conf; redis-server /tmp/redis.conf --dir /tmp --loglevel verbose" , "cpus" : 0.2, "mem" : 256.0, "instances" : 1, "maxLaunchDelaySeconds" : 5, "healthChecks" : [{ "protocol" : "COMMAND" , "command" : { "value" : "env; redis-cli -h $HOST -n 1 info | egrep 'role:master|master_link_status:up'" }, "gracePeriodSeconds" : 20, "intervalSeconds" : 20, "portIndex" : 0, "timeoutSeconds" : 10, "maxConsecutiveFailures" : 3 }] }
          Hide
          sttts Dr. Stefan Schimanski added a comment - - edited

          And Marathon logs of the task on 0.22.1:

          Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,786] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:08 srv004 marathon[1375]: [WARN] [07/23/2015 10:08:08.820] [pool-2-thread-76] [akka://marathon/user/$b] Failed to post MesosStatusUpdateEvent(20150723-081517-100663306-5050-2386-S1,company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,TASK_RUNNING,,/company-master/databases/redis,srv004,Buffer(31578),2015-07-23T10:08:04.839Z,status_update_event,2015-07-23T10:08:08.797Z) to http://10.0.0.4:8000/api/marathon/event_callback WARNING arguments left: 1
          Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,944] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,947] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165)
          Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,947] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:08.947Z)] to health check actor [Actor[akka://marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          Jul 23 10:08:08 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:08.949] [marathon-akka.actor.default-dispatcher-13] [akka://marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:08.947Z)]
          Jul 23 10:08:08 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:08.949] [marathon-akka.actor.default-dispatcher-27] [akka://marathon/user/MarathonScheduler/$a/DeploymentManager/ca1e1c64-7453-4f47-af42-c066c88bbc0a/$a] Killing old task company-master_databases_redis.762a438f-311a-11e5-917e-cee15558ba83 because company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 became reachable
          Jul 23 10:08:08 srv004 marathon[1375]: [WARN] [07/23/2015 10:08:08.970] [pool-2-thread-73] [akka://marathon/user/$b] Failed to post HealthStatusChanged(/company-master/databases/redis,company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,true,health_status_changed_event,2015-07-23T10:08:08.949Z) to http://10.0.0.4:8000/api/marathon/event_callback WARNING arguments left: 1
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,529] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,530] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165)
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,531] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.530Z)] to health check actor [Actor[akka://marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          Jul 23 10:08:12 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:12.531] [marathon-akka.actor.default-dispatcher-21] [akka://marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.530Z)]
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,585] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,587] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165)
          Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,588] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.588Z)] to health check actor [Actor[akka://marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          Jul 23 10:08:12 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:12.588] [marathon-akka.actor.default-dispatcher-18] [akka://marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.588Z)]
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,500] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,501] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165)
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,502] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.501Z)] to health check actor [Actor[akka://marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          Jul 23 10:08:42 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:42.502] [marathon-akka.actor.default-dispatcher-18] [akka://marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.501Z)]
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,537] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99)
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,539] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [true] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165)
          Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,539] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.539Z)] to health check actor [Actor[akka://marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          
          Show
          sttts Dr. Stefan Schimanski added a comment - - edited And Marathon logs of the task on 0.22.1: Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,786] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:08 srv004 marathon[1375]: [WARN] [07/23/2015 10:08:08.820] [pool-2-thread-76] [akka: //marathon/user/$b] Failed to post MesosStatusUpdateEvent(20150723-081517-100663306-5050-2386-S1,company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,TASK_RUNNING,,/company-master/databases/redis,srv004,Buffer(31578),2015-07-23T10:08:04.839Z,status_update_event,2015-07-23T10:08:08.797Z) to http://10.0.0.4:8000/api/marathon/event_callback WARNING arguments left: 1 Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,944] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,947] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [ true ] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165) Jul 23 10:08:08 srv004 marathon[1375]: [2015-07-23 10:08:08,947] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:08.947Z)] to health check actor [Actor[akka: //marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186) Jul 23 10:08:08 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:08.949] [marathon-akka.actor. default -dispatcher-13] [akka: //marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:08.947Z)] Jul 23 10:08:08 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:08.949] [marathon-akka.actor. default -dispatcher-27] [akka: //marathon/user/MarathonScheduler/$a/DeploymentManager/ca1e1c64-7453-4f47-af42-c066c88bbc0a/$a] Killing old task company-master_databases_redis.762a438f-311a-11e5-917e-cee15558ba83 because company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 became reachable Jul 23 10:08:08 srv004 marathon[1375]: [WARN] [07/23/2015 10:08:08.970] [pool-2-thread-73] [akka: //marathon/user/$b] Failed to post HealthStatusChanged(/company-master/databases/redis,company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z, true ,health_status_changed_event,2015-07-23T10:08:08.949Z) to http://10.0.0.4:8000/api/marathon/event_callback WARNING arguments left: 1 Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,529] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,530] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [ true ] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165) Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,531] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.530Z)] to health check actor [Actor[akka: //marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186) Jul 23 10:08:12 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:12.531] [marathon-akka.actor. default -dispatcher-21] [akka: //marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.530Z)] Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,585] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,587] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [ true ] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165) Jul 23 10:08:12 srv004 marathon[1375]: [2015-07-23 10:08:12,588] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.588Z)] to health check actor [Actor[akka: //marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186) Jul 23 10:08:12 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:12.588] [marathon-akka.actor. default -dispatcher-18] [akka: //marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:12.588Z)] Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,500] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,501] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [ true ] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165) Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,502] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.501Z)] to health check actor [Actor[akka: //marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186) Jul 23 10:08:42 srv004 marathon[1375]: [INFO] [07/23/2015 10:08:42.502] [marathon-akka.actor. default -dispatcher-18] [akka: //marathon/user/$Rb] Received health result: [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.501Z)] Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,537] INFO Received status update for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83: TASK_RUNNING (Reconciliation: Latest task state) (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$36c5b88a:99) Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,539] INFO Received status for [company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83] with version [2015-07-23T10:08:04.839Z] and healthy [ true ] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:165) Jul 23 10:08:42 srv004 marathon[1375]: [2015-07-23 10:08:42,539] INFO Forwarding health result [Healthy(company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83,2015-07-23T10:08:04.839Z,2015-07-23T10:08:42.539Z)] to health check actor [Actor[akka: //marathon/user/$Rb#521772329]] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$2f9a72a2:186)
          Hide
          sttts Dr. Stefan Schimanski added a comment - - edited

          Marathon logs of the same task on 0.23:

          mesos-slave.INFO:I0723 09:08:47.880939 40256 slave.cpp:4319] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 09:08:47.883102 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581929482days in the future
          mesos-slave.INFO:I0723 09:08:47.883256 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581894815days in the future
          mesos-slave.INFO:I0723 09:08:47.883311 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581871407days in the future
          mesos-slave.INFO:I0723 09:08:47.883541 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581850667days in the future
          mesos-slave.INFO:I0723 09:08:47.890535 40252 status_update_manager.cpp:205] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.441830  4649 slave.cpp:1244] Got assigned task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.442643  4649 slave.cpp:1355] Launching task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.453456  4649 slave.cpp:4733] Launching executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.457329  4649 slave.cpp:1573] Queuing task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework '20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.462908  4650 docker.cpp:766] Starting container '0597b26e-56b2-42c5-8879-229dc0bcba21' for task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' (and executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e') of framework '20150703-103541-67108874-5050-13727-0003'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.558148  4647 docker.cpp:390] Checkpointing pid 40350 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21/pids/forked.pid'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.616792  4647 slave.cpp:2333] Got registration for executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.619221  4652 slave.cpp:1729] Sending queued task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' to executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.335752  4653 slave.cpp:2671] Handling status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336181  4653 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336634  4653 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342088  4653 slave.cpp:2926] Forwarding the update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.6:5050
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342319  4653 slave.cpp:2856] Sending acknowledgement for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355550  4653 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355825  4653 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          
          Show
          sttts Dr. Stefan Schimanski added a comment - - edited Marathon logs of the same task on 0.23: mesos-slave.INFO:I0723 09:08:47.880939 40256 slave.cpp:4319] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 09:08:47.883102 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581929482days in the future mesos-slave.INFO:I0723 09:08:47.883256 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581894815days in the future mesos-slave.INFO:I0723 09:08:47.883311 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581871407days in the future mesos-slave.INFO:I0723 09:08:47.883541 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581850667days in the future mesos-slave.INFO:I0723 09:08:47.890535 40252 status_update_manager.cpp:205] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.441830 4649 slave.cpp:1244] Got assigned task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.442643 4649 slave.cpp:1355] Launching task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.453456 4649 slave.cpp:4733] Launching executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.457329 4649 slave.cpp:1573] Queuing task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework '20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.462908 4650 docker.cpp:766] Starting container '0597b26e-56b2-42c5-8879-229dc0bcba21' for task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' (and executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e') of framework '20150703-103541-67108874-5050-13727-0003' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.558148 4647 docker.cpp:390] Checkpointing pid 40350 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21/pids/forked.pid' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.616792 4647 slave.cpp:2333] Got registration for executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.619221 4652 slave.cpp:1729] Sending queued task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' to executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.335752 4653 slave.cpp:2671] Handling status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336181 4653 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336634 4653 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342088 4653 slave.cpp:2926] Forwarding the update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.6:5050 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342319 4653 slave.cpp:2856] Sending acknowledgement for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355550 4653 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355825 4653 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          Hide
          sttts Dr. Stefan Schimanski added a comment - - edited

          The mesos-slave logs of a task on 0.22.1:

          The status updates include "in health state healthy" which is not visible in 0.23.

          mesos-slave.INFO:I0723 10:08:08.253509 40252 slave.cpp:1144] Got assigned task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.254894 40252 slave.cpp:1254] Launching task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.265609 40252 slave.cpp:4208] Launching executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8'
          mesos-slave.INFO:I0723 10:08:08.267479 40252 slave.cpp:1401] Queuing task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' for executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework '20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.275472 40252 docker.cpp:626] Starting container '30672d3a-249c-4468-b620-a5873c8a87a8' for task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' (and executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83') of framework '20150703-103541-67108874-5050-13727-0003'
          mesos-slave.INFO:I0723 10:08:08.700194 40250 docker.cpp:277] Checkpointing pid 33342 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8/pids/forked.pid'
          mesos-slave.INFO:I0723 10:08:08.706665 40255 slave.cpp:3165] Monitoring executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework '20150703-103541-67108874-5050-13727-0003' in container '30672d3a-249c-4468-b620-a5873c8a87a8'
          mesos-slave.INFO:I0723 10:08:08.755120 40254 slave.cpp:2164] Got registration for executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687
          mesos-slave.INFO:I0723 10:08:08.757438 40254 slave.cpp:1555] Sending queued task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' to executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.769215 40256 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687
          mesos-slave.INFO:I0723 10:08:08.772658 40256 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.775440 40256 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.781846 40251 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.5:5050
          mesos-slave.INFO:I0723 10:08:08.782343 40251 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:52687
          mesos-slave.INFO:I0723 10:08:08.794363 40255 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.794682 40255 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.934859 40253 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687
          mesos-slave.INFO:I0723 10:08:08.935350 40253 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.935422 40253 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.939698 40255 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.5:5050
          mesos-slave.INFO:I0723 10:08:08.940003 40255 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:52687
          mesos-slave.INFO:I0723 10:08:08.954673 40253 status_update_manager.cpp:389] Received status update acknowledgement (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 10:08:08.955098 40253 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.253509 40252 slave.cpp:1144] Got assigned task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.254894 40252 slave.cpp:1254] Launching task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.265609 40252 slave.cpp:4208] Launching executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8'
          
          Show
          sttts Dr. Stefan Schimanski added a comment - - edited The mesos-slave logs of a task on 0.22.1: The status updates include "in health state healthy" which is not visible in 0.23. mesos-slave.INFO:I0723 10:08:08.253509 40252 slave.cpp:1144] Got assigned task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.254894 40252 slave.cpp:1254] Launching task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.265609 40252 slave.cpp:4208] Launching executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8' mesos-slave.INFO:I0723 10:08:08.267479 40252 slave.cpp:1401] Queuing task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' for executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework '20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.275472 40252 docker.cpp:626] Starting container '30672d3a-249c-4468-b620-a5873c8a87a8' for task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' (and executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83') of framework '20150703-103541-67108874-5050-13727-0003' mesos-slave.INFO:I0723 10:08:08.700194 40250 docker.cpp:277] Checkpointing pid 33342 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8/pids/forked.pid' mesos-slave.INFO:I0723 10:08:08.706665 40255 slave.cpp:3165] Monitoring executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework '20150703-103541-67108874-5050-13727-0003' in container '30672d3a-249c-4468-b620-a5873c8a87a8' mesos-slave.INFO:I0723 10:08:08.755120 40254 slave.cpp:2164] Got registration for executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687 mesos-slave.INFO:I0723 10:08:08.757438 40254 slave.cpp:1555] Sending queued task 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' to executor 'company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.769215 40256 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687 mesos-slave.INFO:I0723 10:08:08.772658 40256 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.775440 40256 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.781846 40251 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.5:5050 mesos-slave.INFO:I0723 10:08:08.782343 40251 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:52687 mesos-slave.INFO:I0723 10:08:08.794363 40255 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.794682 40255 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: 7f5abf5c-702a-4380-9ea5-7a7eb401f3b2) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.934859 40253 slave.cpp:2531] Handling status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:52687 mesos-slave.INFO:I0723 10:08:08.935350 40253 status_update_manager.cpp:317] Received status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.935422 40253 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.939698 40255 slave.cpp:2776] Forwarding the update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.5:5050 mesos-slave.INFO:I0723 10:08:08.940003 40255 slave.cpp:2709] Sending acknowledgement for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:52687 mesos-slave.INFO:I0723 10:08:08.954673 40253 status_update_manager.cpp:389] Received status update acknowledgement (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 10:08:08.955098 40253 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_RUNNING (UUID: fbe6e48f-055e-41f7-88eb-1a076e92c424) for task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 in health state healthy of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.253509 40252 slave.cpp:1144] Got assigned task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.254894 40252 slave.cpp:1254] Launching task company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-090847.40234:I0723 10:08:08.265609 40252 slave.cpp:4208] Launching executor company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83 of framework 20150703-103541-67108874-5050-13727-0003 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.b787889b-3122-11e5-917e-cee15558ba83/runs/30672d3a-249c-4468-b620-a5873c8a87a8'
          Hide
          sttts Dr. Stefan Schimanski added a comment - - edited

          The mesos-slave logs of a task with 0.23:

          mesos-slave.INFO:I0723 09:08:47.880939 40256 slave.cpp:4319] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.INFO:I0723 09:08:47.883102 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581929482days in the future
          mesos-slave.INFO:I0723 09:08:47.883256 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581894815days in the future
          mesos-slave.INFO:I0723 09:08:47.883311 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581871407days in the future
          mesos-slave.INFO:I0723 09:08:47.883541 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581850667days in the future
          mesos-slave.INFO:I0723 09:08:47.890535 40252 status_update_manager.cpp:205] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.441830  4649 slave.cpp:1244] Got assigned task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.442643  4649 slave.cpp:1355] Launching task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.453456  4649 slave.cpp:4733] Launching executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.457329  4649 slave.cpp:1573] Queuing task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework '20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.462908  4650 docker.cpp:766] Starting container '0597b26e-56b2-42c5-8879-229dc0bcba21' for task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' (and executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e') of framework '20150703-103541-67108874-5050-13727-0003'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.558148  4647 docker.cpp:390] Checkpointing pid 40350 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21/pids/forked.pid'
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.616792  4647 slave.cpp:2333] Got registration for executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.619221  4652 slave.cpp:1729] Sending queued task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' to executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.335752  4653 slave.cpp:2671] Handling status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336181  4653 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336634  4653 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342088  4653 slave.cpp:2926] Forwarding the update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.6:5050
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342319  4653 slave.cpp:2856] Sending acknowledgement for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:36558
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355550  4653 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355825  4653 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          
          Show
          sttts Dr. Stefan Schimanski added a comment - - edited The mesos-slave logs of a task with 0.23: mesos-slave.INFO:I0723 09:08:47.880939 40256 slave.cpp:4319] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.INFO:I0723 09:08:47.883102 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581929482days in the future mesos-slave.INFO:I0723 09:08:47.883256 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' for gc 6.99819581894815days in the future mesos-slave.INFO:I0723 09:08:47.883311 40253 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581871407days in the future mesos-slave.INFO:I0723 09:08:47.883541 40253 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for gc 6.99819581850667days in the future mesos-slave.INFO:I0723 09:08:47.890535 40252 status_update_manager.cpp:205] Recovering executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.441830 4649 slave.cpp:1244] Got assigned task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.442643 4649 slave.cpp:1355] Launching task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e for framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.453456 4649 slave.cpp:4733] Launching executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/mesos/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.457329 4649 slave.cpp:1573] Queuing task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' for executor company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework '20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.462908 4650 docker.cpp:766] Starting container '0597b26e-56b2-42c5-8879-229dc0bcba21' for task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' (and executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e') of framework '20150703-103541-67108874-5050-13727-0003' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.558148 4647 docker.cpp:390] Checkpointing pid 40350 to '/tmp/mesos/meta/slaves/20150723-081517-100663306-5050-2386-S1/frameworks/20150703-103541-67108874-5050-13727-0003/executors/company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e/runs/0597b26e-56b2-42c5-8879-229dc0bcba21/pids/forked.pid' mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.616792 4647 slave.cpp:2333] Got registration for executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:03.619221 4652 slave.cpp:1729] Sending queued task 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' to executor 'company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e' of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.335752 4653 slave.cpp:2671] Handling status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 from executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336181 4653 status_update_manager.cpp:322] Received status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.336634 4653 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342088 4653 slave.cpp:2926] Forwarding the update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to master@10.0.0.6:5050 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.342319 4653 slave.cpp:2856] Sending acknowledgement for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 to executor(1)@10.0.0.4:36558 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355550 4653 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003 mesos-slave.srv004.invalid-user.log.INFO.20150723-081530.4292:I0723 08:32:04.355825 4653 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_RUNNING (UUID: a9550037-1716-40a2-be34-973ad3d97ab6) for task company-master_databases_redis.4b6df685-3115-11e5-b6ee-66ed7426526e of framework 20150703-103541-67108874-5050-13727-0003
          Hide
          sttts Dr. Stefan Schimanski added a comment -
          Show
          sttts Dr. Stefan Schimanski added a comment - /cc Peter Kolloch

            People

            • Assignee:
              haosdent@gmail.com haosdent
              Reporter:
              sttts Dr. Stefan Schimanski
              Shepherd:
              Timothy Chen
            • Votes:
              3 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development