Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-6656

Nested containers can become unkillable

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • agent, containerization
    • 0

    Description

      An incident occurred recently in a cluster running a build of Mesos based on commit 757319357471227c0a1e906076eae8f9aa2fdbd6 from master. A task group of five tasks was launched via Marathon. After the tasks were launched, one of the containers quickly exited and was successfully destroyed. A couple minutes later, the task group was killed manually via Marathon, and the agent can then be seen repeatedly attempting to kill the tasks for hours. No calls to WAIT_NESTED_CONTAINER are visible in the agent logs, and the executor logs do not indicate at any point that the nested containers were launched successfully.

      Agent logs:

      Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.890911  6406 slave.cpp:1539] Got assigned task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892299  6406 gc.cpp:83] Unscheduling '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000' from gc
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.892379  6406 gc.cpp:83] Unscheduling '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000' from gc
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893131  6405 slave.cpp:1701] Launching task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.893435  6405 paths.cpp:536] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91' to user 'root'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898026  6405 slave.cpp:6179] Launching executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 with resources cpus(*):0.1; mem(*):32; disk(*):10; ports(*):[21421-21425] in work directory '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.898731  6407 docker.cpp:1000] Skipping non-docker container
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899050  6407 containerizer.cpp:938] Starting container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 for executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.899909  6405 slave.cpp:1987] Queued task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] for executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.907033  6405 memory.cpp:451] Started listening for OOM events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.908336  6405 memory.cpp:562] Started listening on 'low' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.909638  6405 memory.cpp:562] Started listening on 'medium' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.910923  6405 memory.cpp:562] Started listening on 'critical' memory pressure events for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.912464  6405 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 32MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.914640  6405 memory.cpp:251] Updated 'memory.limit_in_bytes' to 32MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.915769  6405 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.917944  6405 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 10ms (cpus 0.1) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918515  6404 isolator_module.cpp:55] Container prepare: container_id[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] container_config[directory: "/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91" user: "root" executor_info { executor_id { value: "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" } command { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } framework_id { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rcluster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } labels { } type: DEFAULT } command_info { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } container_info { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rc
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: luster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" }]
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918647  6404 container_assigner.cpp:64] Registering and retrieving endpoint for container_id[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] executor_info[executor_id { value: "instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001" } resources { name: "cpus" type: SCALAR scalar { value: 0.1 } } resources { name: "mem" type: SCALAR scalar { value: 32 } } resources { name: "disk" type: SCALAR scalar { value: 10 } } resources { name: "ports" type: RANGES ranges { range { begin: 21421 end: 21425 } } role: "*" } command { value: "/opt/mesosphere/packages/mesos--a00089c894f82b6455d1db7c4267a742458020b3/libexec/mesos/mesos-default-executor" user: "root" shell: false arguments: "mesos-default-executor" } framework_id { value: "ce4bd8be-1198-4819-81d4-9a8439439741-0000" } container { type: MESOS network_infos { labels { labels { key: "rcluster.location" value: "globalqa.las2.test0" } labels { key: "rcluster.group" value: "dat" } labels { key: "rcluster.application" value: "scout" } } name: "rcluster-cni" port_mappings { host_port: 21421 container_port: 8080 protocol: "tcp" } port_mappings { host_port: 21422 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21423 container_port: 8093 protocol: "tcp" } port_mappings { host_port: 21424 container_port: 8090 protocol: "tcp" } port_mappings { host_port: 21425 container_port: 8090 protocol: "tcp" } } } labels { } type: DEFAULT].
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918715  6404 sync_util.hpp:39] Dispatching and waiting <=5s for ticket 431: register_and_update_cache
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.918865  6488 container_reader_impl.cpp:34] Reader constructed for 198.51.100.1:0
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919052  6488 container_reader_impl.cpp:103] Reader listening on 198.51.100.1:51243
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919108  6488 container_assigner_strategy.cpp:156] New ephemeral-port reader for container[value: "8750c2a7-8bef-4a69-8ef2-b873f884bf91"] created at endpoint[198.51.100.1:51243].
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919144  6488 container_state_cache_impl.cpp:134] Writing container file[/var/run/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91] with endpoint[198.51.100.1:51243]
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919220  6488 sync_util.hpp:136] Result for ticket 431 complete, returning value.
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.919275  6404 sync_util.hpp:83] Dispatch result obtained for ticket 431 after waiting <=5s: register_and_update_cache
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.924211  6404 systemd.cpp:96] Assigned child process '2701' to 'mesos_executors.slice'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.927724  6404 systemd.cpp:96] Assigned child process '2702' to 'mesos_executors.slice'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.929244  6403 linux_launcher.cpp:421] Launching container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 and cloning with namespaces CLONE_NEWNS | CLONE_NEWUTS | CLONE_NEWNET
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.935976  6403 systemd.cpp:96] Assigned child process '2705' to 'mesos_executors.slice'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.943640  6409 containerizer.cpp:1489] Checkpointing container's forked pid 2705 to '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/pids/forked.pid'
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:16.945247  6410 cni.cpp:814] Bind mounted '/proc/2705/ns/net' to '/run/mesos/isolators/network/cni/8750c2a7-8bef-4a69-8ef2-b873f884bf91/ns' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.961654  2701 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:16 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:16.963691  2702 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.250937  6410 cni.cpp:1230] Got assigned IPv4 address '10.47.11.252/32' from CNI network 'rcluster-cni' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.251171  6410 cni.cpp:939] DNS nameservers for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 are:
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.22
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: nameserver 10.190.4.23
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942303  6404 http.cpp:277] HTTP POST for /slave(1)/api/v1/executor from 10.47.11.252:38416
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942446  6404 slave.cpp:3022] Received Subscribe request for HTTP executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.942533  6404 slave.cpp:3085] Creating a marker file for HTTP based executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP) at path '/var/lib/mesos/slave/meta/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/http.marker'
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.944046  6404 disk.cpp:207] Updating the disk resources for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91 to cpus(*):0.6; mem(*):192; disk(*):170; ports(*):[21421-21425]
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.945669  6407 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 192MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.947779  6407 memory.cpp:251] Updated 'memory.limit_in_bytes' to 192MB for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.948951  6407 cpu.cpp:101] Updated 'cpu.shares' to 614 (cpus 0.6) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.951117  6407 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 60ms (cpus 0.6) for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91
Nov 29 04:04:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:17.952219  6404 slave.cpp:2220] Sending queued task group task group containing tasks [ dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4, dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 ] to executor 'instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001' of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000 (via HTTP)
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002063  6408 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.002223  6408 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003235  6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003327  6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003432  6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707' to user 'nobody'
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.003465  6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.005939  6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006057  6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006265  6407 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.006384  6407 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.008982  6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.009096  6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b' to user 'nobody'
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013618  6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.013708  6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668' to user 'nobody'
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018230  6409 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.018399  6409 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47' to user 'nobody'
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.022992  6403 http.cpp:277] HTTP POST for /slave(1)/api/v1 from 10.47.11.252:38420
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023133  6403 http.cpp:353] Processing call LAUNCH_NESTED_CONTAINER
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023598  6406 containerizer.cpp:1657] Starting nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.023705  6406 containerizer.cpp:1681] Trying to chown '/var/lib/mesos/slave/slaves/ce4bd8be-1198-4819-81d4-9a8439439741-S1/frameworks/ce4bd8be-1198-4819-81d4-9a8439439741-0000/executors/instance-dat_scout.e57be1fe-b5e8-11e6-995b-70b3d5800001/runs/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c' to user 'nobody'
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462172  6405 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/063dbcd6-1a0a-4058-a615-7e6deaf17707/backends/copy/rootfses/97f1a1df-1eec-46e3-8a93-8a2b80d48dd2' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.462401  6405 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/a3b19108-d56c-44ea-b20a-912d539dde7b/backends/copy/rootfses/b93ffe07-6f64-42e1-b8c4-77bb224fc9e0' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.465819  6410 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/02c64ae4-ccdd-415d-946a-3f665acbb668/backends/copy/rootfses/5d388325-8b4d-496f-b520-4a96584a5971' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.470854  6409 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c5c10cff-a6af-4e51-911e-61d04e5c1d47/backends/copy/rootfses/5b4d32da-7d82-45d2-ad28-85a48456a18b' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.471132  6409 provisioner.cpp:294] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:18.893782  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.164958  6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.171018  6405 systemd.cpp:96] Assigned child process '2880' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.176822  6405 systemd.cpp:96] Assigned child process '2881' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.179283  6405 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.063dbcd6-1a0a-4058-a615-7e6deaf17707 and cloning with namespaces CLONE_NEWNS
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.185796  6409 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.200559  6405 systemd.cpp:96] Assigned child process '2884' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.209010  2881 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.211642  2880 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.215493  6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.228160  6407 systemd.cpp:96] Assigned child process '2896' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.234012  6407 systemd.cpp:96] Assigned child process '2897' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.239496  6409 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c5c10cff-a6af-4e51-911e-61d04e5c1d47 and cloning with namespaces CLONE_NEWNS
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.242473  6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.243351  6405 runtime.cpp:109] Container user 'nobody' is not supported yet for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.254703  6409 systemd.cpp:96] Assigned child process '2900' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.268317  2896 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.274961  6405 systemd.cpp:96] Assigned child process '2902' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.279268  6405 systemd.cpp:96] Assigned child process '2911' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.284328  6405 systemd.cpp:96] Assigned child process '2912' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.286672  6404 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.a3b19108-d56c-44ea-b20a-912d539dde7b and cloning with namespaces CLONE_NEWNS
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.299939  6405 systemd.cpp:96] Assigned child process '2914' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.304103  6405 systemd.cpp:96] Assigned child process '2917' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.307452  6405 systemd.cpp:96] Assigned child process '2918' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.309020  2897 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.311029  6404 systemd.cpp:96] Assigned child process '2916' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.331065  2911 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.333595  6405 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.02c64ae4-ccdd-415d-946a-3f665acbb668 and cloning with namespaces CLONE_NEWNS
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.342017  2902 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.372247  6405 systemd.cpp:96] Assigned child process '2937' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.372742  2912 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.378247  2914 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.388391  2917 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.388517  6410 linux_launcher.cpp:421] Launching nested container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c and cloning with namespaces CLONE_NEWNS
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: WARNING: Logging before InitGoogleLogging() is written to STDERR
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: W1129 04:04:20.396700  2918 process.cpp:882] Failed SSL connections will be downgraded to a non-SSL socket
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.400269  6410 systemd.cpp:96] Assigned child process '2969' to 'mesos_executors.slice'
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772186  6403 containerizer.cpp:2313] Container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c has exited
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772228  6403 containerizer.cpp:1950] Destroying container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c in RUNNING state
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.772330  6403 linux_launcher.cpp:498] Asked to destroy container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.773825  6403 linux_launcher.cpp:541] Using freezer to destroy cgroup mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.776217  6407 cgroups.cpp:2705] Freezing cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.780053  6410 cgroups.cpp:1439] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c after 3.798016ms
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.783502  6407 cgroups.cpp:2723] Thawing cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.786717  6407 cgroups.cpp:1468] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/8750c2a7-8bef-4a69-8ef2-b873f884bf91/mesos/c8f6dc94-baba-44be-ab65-49538645a31c after 3.186176ms
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.791746  6404 provisioner.cpp:488] Destroying container rootfs at '/var/lib/mesos/slave/provisioner/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/backends/copy/rootfses/244e36f5-ecca-444f-9bf6-05aa77cb7263' for container 8750c2a7-8bef-4a69-8ef2-b873f884bf91.c8f6dc94-baba-44be-ab65-49538645a31c
Nov 29 04:04:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:20.873241  6405 containerizer.cpp:2229] Checkpointing termination state to nested container's runtime directory '/var/run/mesos/containers/8750c2a7-8bef-4a69-8ef2-b873f884bf91/containers/c8f6dc94-baba-44be-ab65-49538645a31c/termination'
Nov 29 04:04:21 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:21.153594  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:23.410384  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:25.667546  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:27.925597  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:30 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:30.198536  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:32.456645  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:34.716507  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:36.974459  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:39 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:39.229652  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:41 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:41.482465  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:43 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:43.736668  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.994588  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:45.999001  6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=0, failed=0, pending=0 (state CONNECT_PENDING)
Nov 29 04:04:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:46.187460  6408 slave.cpp:5044] Current disk usage 10.89%. Max allowed age: 1.582110178819861days
Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.161314  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57362 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:48 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:48.250435  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033789  6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_PENDING)
Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.033946  6488 metrics_tcp_sender.cpp:112] Attempting to open connection to 127.0.0.1:8124
Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034059  6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111) when connecting to metrics service at 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
Nov 29 04:04:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:49.034081  6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in 60s...
Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.161448  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57426 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:50 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:50.503459  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.161912  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57486 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:52 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:52.765455  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:54 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:54.161412  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57546 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:55 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:55.022529  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:56.163820  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57606 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:57 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:57.279559  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:04:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:58.162578  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57666 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:04:59 ip-10-190-112-199 mesos-agent[6397]: I1129 04:04:59.538532  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:00.161715  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57726 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:01 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:01.801764  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:02.160838  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57790 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.060504  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:04 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:04.162878  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57908 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.161875  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:57962 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:06 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:06.318395  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.160797  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58016 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:08 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:08.572476  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.162160  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58070 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:10 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:10.828313  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:12 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:12.161572  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58124 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:13 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:13.084312  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:14.164269  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58182 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:15 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:15.339344  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.165223  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58238 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:16.919145  6488 container_reader_impl.cpp:146] Throughput from container at port 51243 (bytes): received=0, throttled=0
Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.476977  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.74:58298 with User-Agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36' with X-Forwarded-For='10.192.128.26'
Nov 29 04:05:17 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:17.594532  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:19 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:19.856494  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:22 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:22.112598  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:24 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:24.371556  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:26.628479  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:28 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:28.885504  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:31 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:31.143674  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:33 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:33.434650  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:35 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:35.694718  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:37 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:37.952450  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:40 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:40.209468  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:42 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:42.466473  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:44 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:44.724468  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:45 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:45.999135  6488 metrics_tcp_sender.cpp:252] TCP Throughput (bytes): sent=0, dropped=373, failed=0, pending=0 (state CONNECT_PENDING)
Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.188647  6408 slave.cpp:5044] Current disk usage 10.83%. Max allowed age: 1.583336138877014days
Nov 29 04:05:46 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:46.986548  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034080  6488 metrics_tcp_sender.cpp:127] Timed out when opening metrics connection to 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_PENDING)
Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034231  6488 metrics_tcp_sender.cpp:112] Attempting to open connection to 127.0.0.1:8124
Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034332  6488 metrics_tcp_sender.cpp:140] Got error 'Connection refused'(system:111) when connecting to metrics service at 127.0.0.1:8124. This is expected if no Metrics Collector is running on this agent. (state CONNECT_IN_PROGRESS)
Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.034358  6488 metrics_tcp_sender.cpp:94] Scheduling reconnect to 127.0.0.1:8124 in 60s...
Nov 29 04:05:49 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:49.243544  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:51 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:51.500578  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:53 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:53.755667  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:56 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:56.010568  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:05:58 ip-10-190-112-199 mesos-agent[6397]: I1129 04:05:58.269441  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:00 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:00.526623  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:02 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:02.784615  6406 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:05 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:05.043455  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:07 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:07.299536  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:09 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:09.556404  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:11 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:11.813442  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:14 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:14.068482  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.325479  6410 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:16 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:16.919404  6488 container_reader_impl.cpp:146] Throughput from container at port 51243 (bytes): received=0, throttled=0
Nov 29 04:06:18 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:18.581571  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:20 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:20.841563  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:23 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:23.098562  6403 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:25 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:25.358496  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886557  6406 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886762  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886831  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886884  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:26 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:26.886924  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:27 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:27.615567  6404 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:29 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:29.877619  6405 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:32 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:32.135524  6408 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:34 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:34.402644  6409 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.664479  6407 http.cpp:277] HTTP GET for /slave(1)/state from 10.190.112.199:36156
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903115  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903259  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903317  6405 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903436  6408 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000
Nov 29 04:06:36 ip-10-190-112-199 mesos-agent[6397]: I1129 04:06:36.903501  6408 slave.cpp:2288] Asked to kill task dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2 of framework ce4bd8be-1198-4819-81d4-9a8439439741-0000

      Executor log:

      I1129 04:04:17.832391  2734 executor.cpp:189] Version: 1.1.0
I1129 04:04:17.956600  2738 default_executor.cpp:123] Received SUBSCRIBED event
I1129 04:04:17.956832  2738 default_executor.cpp:127] Subscribed executor on 10.190.112.199
I1129 04:04:17.968544  2735 default_executor.cpp:123] Received LAUNCH_GROUP event
I1129 04:06:26.899070  2735 default_executor.cpp:123] Received KILL event
I1129 04:06:26.899101  2735 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
W1129 04:06:26.899122  2735 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it is no longer active
I1129 04:06:26.899235  2742 default_executor.cpp:123] Received KILL event
I1129 04:06:26.899251  2742 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
W1129 04:06:26.899260  2742 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it is no longer active
I1129 04:06:26.899425  2741 default_executor.cpp:123] Received KILL event
I1129 04:06:26.899443  2741 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
W1129 04:06:26.899448  2741 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it is no longer active
I1129 04:06:26.899591  2740 default_executor.cpp:123] Received KILL event
I1129 04:06:26.899616  2740 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
W1129 04:06:26.899627  2740 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it is no longer active
I1129 04:06:26.899699  2735 default_executor.cpp:123] Received KILL event
I1129 04:06:26.899713  2735 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
W1129 04:06:26.899721  2735 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it is no longer active
I1129 04:06:36.915755  2740 default_executor.cpp:123] Received KILL event
I1129 04:06:36.915786  2740 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5'
W1129 04:06:36.915796  2740 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server5' as it is no longer active
I1129 04:06:36.915915  2736 default_executor.cpp:123] Received KILL event
I1129 04:06:36.915931  2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1'
W1129 04:06:36.915940  2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server1' as it is no longer active
I1129 04:06:36.915962  2736 default_executor.cpp:123] Received KILL event
I1129 04:06:36.915972  2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3'
W1129 04:06:36.915979  2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server3' as it is no longer active
I1129 04:06:36.915997  2736 default_executor.cpp:123] Received KILL event
I1129 04:06:36.916007  2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4'
W1129 04:06:36.916013  2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server4' as it is no longer active
I1129 04:06:36.916031  2736 default_executor.cpp:123] Received KILL event
I1129 04:06:36.916040  2736 default_executor.cpp:802] Received kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2'
W1129 04:06:36.916048  2736 default_executor.cpp:813] Ignoring kill for task 'dat_scout.instance-e57be1fe-b5e8-11e6-995b-70b3d5800001.scout-server2' as it is no longer active

continuing for hours ...

      Meanwhile, the tasks show up as STAGING in the Mesos web UI, and their sandboxes are empty.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. MESOS-7858 Launching a nested container with namespace/pid isolation, with glibc < 2.25, may deadlock the LinuxLauncher and MesosContainerizer

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              greggomann Greg Mann
              greggomann Greg Mann
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: