Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-4899

Mesos slave crash after killing docker container

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Not A Bug
    • 0.27.1
    • None
    • None
    • None

    Description

      I have experienced an issue where a Mesos slave crashed because a docker task could not be killed properly.
      I'm using marathon to launch the task.

      The setup is the following:

      OS verison: Centos 7.2
      Docker version: 1.8.2
      Mesos-slave: 0.27.1
      Mesos-master: 0.27.1
      Marathon 0.15.3

      The mesos slave crashed (which is not great). This is the log from the mesos slave (both mesos slave had the same issue):

      Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.441756 30870 slave.cpp:1890] Asked to kill task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8
-d7536e319ea2-0000
Mar 07 15:25:16 marathon_mesos-2 docker[13019]: time="2016-03-07T15:25:16.553925302Z" level=info msg="POST /v1.20/containers/mesos-d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145.104860a3-4630-4cbb-8e68-87ecca13fcad/s
top?t=0"
Mar 07 15:25:16 marathon_mesos-2 docker[13019]: time="2016-03-07T15:25:16.559932661Z" level=info msg="Container fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a failed to exit within 0 seconds of
 SIGTERM - using the force"
Mar 07 15:25:16 marathon_mesos-2 systemd[1]: Stopped docker container fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a.
Mar 07 15:25:16 marathon_mesos-2 systemd[1]: Stopping docker container fe412634ec92bb641a18b4c48d399895f703af29492804b927943646bd81ab8a.
Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered disabled state
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn>  (vethd55899a): failed to find device 106 'vethd55899a' with udev
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info>  (vethd55899a): new Veth device (carrier: OFF, driver: 'veth', ifindex: 106)
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info>  (vetha772033): link disconnected
Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered disabled state
Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing address record for fe80::b06a:2fff:fecc:87f1 on vetha772033.
Mar 07 15:25:16 marathon_mesos-2 kernel: device vetha772033 left promiscuous mode
Mar 07 15:25:16 marathon_mesos-2 kernel: docker0: port 4(vetha772033) entered disabled state
Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing workstation service for vethd55899a.
Mar 07 15:25:16 marathon_mesos-2 avahi-daemon[11714]: Withdrawing workstation service for vetha772033.
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn>  (vethd55899a): failed to disable userspace IPv6LL address handling
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info>  (docker0): bridge port vetha772033 was detached
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <info>  (vetha772033): released from master docker0
Mar 07 15:25:16 marathon_mesos-2 NetworkManager[788]: <warn>  (vetha772033): failed to disable userspace IPv6LL address handling
Mar 07 15:25:16 marathon_mesos-2 kernel: XFS (dm-8): Unmounting Filesystem
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.879240 30868 slave.cpp:3001] Handling status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47
c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from executor(1)@XXX.XXX.XXX.XXX:57543
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: E0307 15:25:16.880144 30868 slave.cpp:3205] Failed to update resources for container 104860a3-4630-4cbb-8e68-87ecca13fcad of executor 'giulio.httpd.test.tag.a
47c55be-db01-11e5-a92a-0242eb705eb2' running task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 on status update for terminal task, destroying container: Failed to determine cgroup for the 'cpu' sub
system: Failed to read /proc/4390/cgroup: Failed to open file '/proc/4390/cgroup': No such file or directory
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880388 30868 status_update_manager.cpp:320] Received status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.htt
pd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880416 30870 docker.cpp:1455] Destroying container '104860a3-4630-4cbb-8e68-87ecca13fcad'
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880487 30870 docker.cpp:1515] Sending SIGTERM to executor with pid: 4318
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.880437 30868 status_update_manager.cpp:824] Checkpointing UPDATE for status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.899090 30868 slave.cpp:3353] Forwarding the update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 to master@XXX.XXX.XXX.XXX:5051
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.899364 30868 slave.cpp:3263] Sending acknowledgement for status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 to executor(1)@XXX.XXX.XXX.XXX:57543
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.923931 30867 status_update_manager.cpp:392] Received status update acknowledgement (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.924121 30867 status_update_manager.cpp:824] Checkpointing ACK for status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.950709 30870 docker.cpp:1557] Running docker stop on container '104860a3-4630-4cbb-8e68-87ecca13fcad'
Mar 07 15:25:16 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:16.955036 30870 slave.cpp:3481] executor(1)@XXX.XXX.XXX.XXX:57543 exited
Mar 07 15:25:17 marathon_mesos-2 docker[13019]: time="2016-03-07T15:25:17.003800525Z" level=info msg="POST /v1.20/containers/mesos-d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145.104860a3-4630-4cbb-8e68-87ecca13fcad/stop?t=0"
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.042575 30872 docker.cpp:1654] Executor for container '104860a3-4630-4cbb-8e68-87ecca13fcad' has exited
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.043073 30868 slave.cpp:3816] Executor 'giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2' of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 terminated with signal Terminated
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: I0307 15:25:17.043175 30868 slave.cpp:3920] Cleaning up executor 'giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2' of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 at executor(1)@XXX.XXX.XXX.XXX:57543
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: F0307 15:25:17.043442 30868 slave.cpp:3945] CHECK_SOME(os::touch(path)): Failed to open file: No such file or directory
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]: *** Check failure stack trace: ***
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f73133d  google::LogMessage::Fail()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f73307c  google::LogMessage::SendToLog()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f730f2c  google::LogMessage::Flush()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f733979  google::LogMessageFatal::~LogMessageFatal()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f0faf1c  mesos::internal::slave::Slave::removeExecutor()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f112970  mesos::internal::slave::Slave::executorTerminated()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f6dee21  process::ProcessManager::resume()
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5f6df127  _ZNSt6thread5_ImplISt12_Bind_simpleIFSt5_BindIFZN7process14ProcessManager12init_threadsEvEUlRKSt11atomic_boolE_St17reference_wrapperIS6_EEEvEEE6_M_runEv
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5e023220  (unknown)
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5e27ddc5  start_thread
Mar 07 15:25:17 marathon_mesos-2 mesos-slave[30866]:     @     0x7f4a5da8d28d  __clone
Mar 07 15:25:17 marathon_mesos-2 systemd[1]: mesos-slave.service: main process exited, code=killed, status=6/ABRT
Mar 07 15:25:17 marathon_mesos-2 systemd[1]: Unit mesos-slave.service entered failed state.
Mar 07 15:25:17 marathon_mesos-2 systemd[1]: mesos-slave.service failed.
Mar 07 15:25:17 marathon_mesos-2 ntpd[728]: Deleting interface #42 vetha772033, fe80::b06a:2fff:fecc:87f1#123, interface stats: received=0, sent=0, dropped=0, active_time=1040732 secs
Mar 07 15:25:37 marathon_mesos-2 systemd[1]: mesos-slave.service holdoff time over, scheduling restart.
Mar 07 15:25:37 marathon_mesos-2 systemd[1]: Started Mesos Slave.
Mar 07 15:25:37 marathon_mesos-2 systemd[1]: Starting Mesos Slave...

      Mesos master log

      I0307 15:25:16.443166     8 master.cpp:3855] Telling slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com) to kill task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663
I0307 15:25:16.443385     8 master.cpp:3855] Telling slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com) to kill task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663
I0307 15:25:16.443476     8 master.cpp:3855] Telling slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com) to kill task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663
I0307 15:25:16.903249     8 master.cpp:4789] Status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com)
I0307 15:25:16.903378     8 master.cpp:4837] Forwarding status update TASK_KILLED (UUID: 1b9446db-dcdc-47a7-aa05-0372e84e1b4d) for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
I0307 15:25:16.903561     8 master.cpp:6445] Updating the state of task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0307 15:25:16.924135     8 master.cpp:3947] Processing ACKNOWLEDGE call 1b9446db-dcdc-47a7-aa05-0372e84e1b4d for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145
I0307 15:25:16.924257     8 master.cpp:6511] Removing task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 with resources cpus(*):0.04; mem(*):32; ports(*):[31511-31511] of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com)
I0307 15:25:17.059509     8 master.cpp:4789] Status update TASK_KILLED (UUID: 6ab5a70a-9690-43df-8d05-f4ea08803ac2) for task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.059617     8 master.cpp:4837] Forwarding status update TASK_KILLED (UUID: 6ab5a70a-9690-43df-8d05-f4ea08803ac2) for task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
I0307 15:25:17.059797     8 master.cpp:6445] Updating the state of task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0307 15:25:17.073132     8 master.cpp:3947] Processing ACKNOWLEDGE call 6ab5a70a-9690-43df-8d05-f4ea08803ac2 for task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146
I0307 15:25:17.073266     8 master.cpp:6511] Removing task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 with resources cpus(*):0.04; mem(*):32; ports(*):[31833-31833] of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.088891     8 master.cpp:1172] Slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com) disconnected
I0307 15:25:17.088999     8 master.cpp:2633] Disconnecting slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com)
I0307 15:25:17.089064     8 master.cpp:2652] Deactivating slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-2.domain.com)
I0307 15:25:17.089191     8 hierarchical.cpp:560] Slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S145 deactivated
I0307 15:25:17.121345     8 master.cpp:4789] Status update TASK_KILLED (UUID: 582f00d2-037c-43a1-88c1-2f610717a5d5) for task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 from slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.121451     8 master.cpp:4837] Forwarding status update TASK_KILLED (UUID: 582f00d2-037c-43a1-88c1-2f610717a5d5) for task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000
I0307 15:25:17.121590     8 master.cpp:6445] Updating the state of task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED)
I0307 15:25:17.145149     8 master.cpp:3947] Processing ACKNOWLEDGE call 582f00d2-037c-43a1-88c1-2f610717a5d5 for task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 (marathon) at scheduler-2582f6fa-94eb-4044-9b0b-fa8988a0166b@XXX.XXX.XXX.XXX:55663 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146
I0307 15:25:17.145256     8 master.cpp:6511] Removing task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 with resources cpus(*):0.04; mem(*):32; ports(*):[31167-31167] of framework ef1354df-7ecc-41ac-82d8-d7536e319ea2-0000 on slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.287336     8 master.cpp:1172] Slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com) disconnected
I0307 15:25:17.287485     8 master.cpp:2633] Disconnecting slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.287531     8 master.cpp:2652] Deactivating slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 at slave(1)@XXX.XXX.XXX.XXX:5051 (marathon_mesos-1.domain.com)
I0307 15:25:17.287618     8 hierarchical.cpp:560] Slave d0f20f55-bc6e-43e8-babb-250b0176f5f6-S146 deactivated

      Marathon framework log:

      [2016-03-07 15:25:16,366] INFO Upgrade group id:/ version:2016-03-07T15:25:16.366Z with force:true (mesosphere.marathon.state.GroupManager:marathon-akka.actor.default-dispatcher-46207)
[2016-03-07 15:25:16,368] INFO Computed new deployment plan:
[2016-03-07 15:25:16,368] INFO Deploy plan with force=true:
[2016-03-07 15:25:16,374] INFO [/giulio.httpd.test.tag] expunging app (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-5)
[2016-03-07 15:25:16,458] INFO Updated groups/apps according to deployment plan bc0e6fa3-f1cc-4658-9787-efe85e957869 (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-5)
[2016-03-07 15:25:16,459] INFO Deployment acknowledged. Waiting to get processed:
[2016-03-07 15:25:16,690] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16 +0000] "DELETE //XXX.XXX.XXX.XXX:8080/v2/apps//giulio.httpd.test.tag HTTP/1.1" 200 92 "http://XXX.XXX.XXX.XXX:8080/ui/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0" 32 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-249367)
[2016-03-07 15:25:16,863] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16 +0000] "GET //XXX.XXX.XXX.XXX:8080/ui/img/marathon-favicon.ico HTTP/1.1" 200 32038 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-143659)
[2016-03-07 15:25:16,904] INFO Received status update for task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2: TASK_KILLED () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63362)
[2016-03-07 15:25:16,910] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16 +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/deployments HTTP/1.1" 200 308 "http://XXX.XXX.XXX.XXX:8080/ui/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-123617)
[2016-03-07 15:25:16,912] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16 +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/apps HTTP/1.1" 200 4360 "http://XXX.XXX.XXX.XXX:8080/ui/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-116139)
[2016-03-07 15:25:16,912] INFO XXX.XXX.XXX.XXX - - [07/Mar/2016:15:25:16 +0000] "GET //XXX.XXX.XXX.XXX:8080/v2/queue HTTP/1.1" 200 12 "http://XXX.XXX.XXX.XXX:8080/ui/" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0"  (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$562bf4ff:qtp1361931872-143659)
[2016-03-07 15:25:16,921] INFO Sending event notification for task [giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2] of app [/giulio.httpd.test.tag]: TASK_KILLED (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-25)
[2016-03-07 15:25:16,921] INFO initiating a scale check for app [/giulio.httpd.test.tag] after task [giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2}] terminated (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-25)
[2016-03-07 15:25:16,922] INFO Task giulio.httpd.test.tag.a47c55be-db01-11e5-a92a-0242eb705eb2 has been killed. Waiting for 2 more tasks to be killed. (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46201)
[2016-03-07 15:25:17,060] INFO Received status update for task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2: TASK_KILLED () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63363)
[2016-03-07 15:25:17,070] INFO Sending event notification for task [giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2] of app [/giulio.httpd.test.tag]: TASK_KILLED (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-5)
[2016-03-07 15:25:17,071] INFO initiating a scale check for app [/giulio.httpd.test.tag] after task [giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2}] terminated (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-5)
[2016-03-07 15:25:17,071] INFO Task giulio.httpd.test.tag.a47d401f-db01-11e5-a92a-0242eb705eb2 has been killed. Waiting for 1 more tasks to be killed. (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46207)
[2016-03-07 15:25:17,122] INFO Received status update for task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2: TASK_KILLED () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$4304ca77:Thread-63364)
[2016-03-07 15:25:17,132] INFO Removed app [/giulio.httpd.test.tag] from tracker (mesosphere.marathon.core.task.tracker.TaskTracker$TasksByApp$:marathon-akka.actor.default-dispatcher-46161)
[2016-03-07 15:25:17,133] INFO Sending event notification for task [giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2] of app [/giulio.httpd.test.tag]: TASK_KILLED (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$8fcb1905:ForkJoinPool-2-worker-11)
[2016-03-07 15:25:17,133] INFO initiating a scale check for app [/giulio.httpd.test.tag] after task [giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2}] terminated (mesosphere.marathon.core.task.update.impl.steps.ScaleAppUpdateStepImpl$$EnhancerByGuice$$fc6d4908:ForkJoinPool-2-worker-11)
[2016-03-07 15:25:17,134] INFO Task giulio.httpd.test.tag.a47c079d-db01-11e5-a92a-0242eb705eb2 has been killed. Waiting for 0 more tasks to be killed. (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46195)
[2016-03-07 15:25:17,134] INFO Successfully killed all the tasks (mesosphere.marathon.upgrade.AppStopActor:marathon-akka.actor.default-dispatcher-46195)
[2016-03-07 15:25:17,184] INFO Stopping app /giulio.httpd.test.tag (mesosphere.marathon.SchedulerActions:marathon-akka.actor.default-dispatcher-46195)
[2016-03-07 15:25:17,188] INFO Deployment of / successful (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-46161)
[2016-03-07 15:25:17,188] INFO Removing bc0e6fa3-f1cc-4658-9787-efe85e957869 from list of running deployments (mesosphere.marathon.upgrade.DeploymentManager:marathon-akka.actor.default-dispat

      Attachments

        Issue Links

        duplicates

        Bug - A problem which impairs or prevents the functions of the product. MESOS-4827 Destroy Docker container crashes Mesos slave

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. MESOS-4996 'containerizer->update' will always fail after killing a docker container.

        • Major - Major loss of function.
        • Resolved
        relates to

        Bug - A problem which impairs or prevents the functions of the product. MESOS-5064 Remove default value for the agent `work_dir`

        • Blocker - Blocks development and/or testing work, production could not run
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            baraldi86 Giulio D'Ippolito
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment