Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9180

tasks get stuck in TASK_KILLING on the default executor

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • 1.6.1
    • None
    • executor

    • Ubuntu 18.04, Ubuntu 16.04

    Description

      during our load tests tasks get stuck in TASK_KILLING state

      I0823 16:30:20.367563 21608 executor.cpp:192] Version: 1.6.1
I0823 16:30:20.439478 21684 default_executor.cpp:202] Received SUBSCRIBED event
I0823 16:30:20.441012 21684 default_executor.cpp:206] Subscribed executor on XX.XXX.XX.XXX
I0823 16:30:20.916216 21665 default_executor.cpp:202] Received LAUNCH_GROUP event
I0823 16:30:20.917373 21645 default_executor.cpp:426] Setting 'MESOS_CONTAINER_IP' to: 172.26.10.222
I0823 16:30:22.573794 21658 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:22.575518 21637 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:22.577137 21665 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:33.091509 21642 default_executor.cpp:661] Finished launching tasks [ test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery ] in child containers [ 3680beff-96d2-4ebd-832c-9cbbddf8c507.8e04f74f-cb8b-46b9-8758-340455a844c8, 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7, 3680beff-96d2-4ebd-832c-9cbbddf8c507.ab481072-c8ab-4a76-be8b-7f4431220e7b ]
I0823 16:30:33.091567 21642 default_executor.cpp:685] Waiting on child containers of tasks [ test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery ]
I0823 16:30:33.096014 21647 default_executor.cpp:746] Waiting for child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.8e04f74f-cb8b-46b9-8758-340455a844c8 of task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka'
I0823 16:30:33.096310 21647 default_executor.cpp:746] Waiting for child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 of task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis'
I0823 16:30:33.096470 21647 default_executor.cpp:746] Waiting for child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.ab481072-c8ab-4a76-be8b-7f4431220e7b of task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery'
I0823 16:30:33.521510 21648 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:33.522073 21652 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:33.523569 21679 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:38.593736 21668 checker_process.cpp:814] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' (stdout):
0
PONG
I0823 16:30:38.593777 21668 checker_process.cpp:817] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' (stderr):
I0823 16:30:38.610167 21650 checker_process.cpp:814] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' (stdout):
I0823 16:30:38.610194 21650 checker_process.cpp:817] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' (stderr):
I0823 16:30:38.700561 21681 checker_process.cpp:814] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' (stdout):
I0823 16:30:38.700598 21681 checker_process.cpp:817] Output of the COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' (stderr):
I0823 16:30:42.786908 21649 checker_process.cpp:971] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' returned: 0
I0823 16:30:42.787267 21649 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis', task is healthy
I0823 16:30:45.156363 21658 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:48.454120 21653 checker_process.cpp:971] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' returned: 1
W0823 16:30:48.454218 21653 health_checker.cpp:283] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed: Command terminated with signal Hangup
W0823 16:30:48.454242 21653 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed 1 times consecutively
I0823 16:30:48.454370 21653 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka', task is not healthy
I0823 16:30:50.887114 21666 checker_process.cpp:971] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' returned: 1
W0823 16:30:50.887183 21666 health_checker.cpp:283] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed: Command terminated with signal Hangup
W0823 16:30:50.887198 21666 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed 1 times consecutively
I0823 16:30:50.887295 21657 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery', task is not healthy
I0823 16:30:51.289993 21689 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:30:51.607558 21659 default_executor.cpp:202] Received ACKNOWLEDGED event
W0823 16:31:23.851263 21657 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed: Command timed out after 5secs
W0823 16:31:23.851332 21657 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed 2 times consecutively
I0823 16:31:23.851519 21641 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery', task is not healthy
W0823 16:31:24.081169 21654 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed: Command timed out after 5secs
W0823 16:31:24.081220 21654 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed 2 times consecutively
I0823 16:31:24.081336 21654 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka', task is not healthy
I0823 16:31:24.487970 21659 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:31:26.176144 21682 default_executor.cpp:202] Received ACKNOWLEDGED event
W0823 16:31:48.187378 21659 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed: Command timed out after 5secs
W0823 16:31:48.187428 21659 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed 3 times consecutively
I0823 16:31:48.187537 21659 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery', task is not healthy
W0823 16:31:48.210490 21676 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed: Command timed out after 5secs
W0823 16:31:48.210537 21676 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed 3 times consecutively
I0823 16:31:48.210651 21676 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka', task is not healthy
I0823 16:31:48.426265 21660 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:31:48.427875 21640 default_executor.cpp:202] Received ACKNOWLEDGED event
W0823 16:32:24.028173 21638 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed: Command timed out after 5secs
W0823 16:32:24.028211 21638 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed 4 times consecutively
I0823 16:32:24.028343 21638 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery', task is not healthy
W0823 16:32:24.080215 21688 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed: Command timed out after 5secs
W0823 16:32:24.080267 21688 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed 4 times consecutively
I0823 16:32:24.080369 21688 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka', task is not healthy
I0823 16:32:24.994634 21672 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:25.002722 21683 default_executor.cpp:202] Received ACKNOWLEDGED event
W0823 16:32:49.181438 21671 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed: Command timed out after 5secs
W0823 16:32:49.181476 21671 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' failed 5 times consecutively
I0823 16:32:49.181608 21671 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery', task is not healthy
I0823 16:32:49.182938 21671 default_executor.cpp:1249] Received kill for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery'
I0823 16:32:49.183014 21671 checker_process.cpp:281] Stopped COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery'
I0823 16:32:49.183149 21671 default_executor.cpp:1124] Killing task test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.ab481072-c8ab-4a76-be8b-7f4431220e7b with SIGTERM signal
I0823 16:32:49.183159 21671 default_executor.cpp:1135] Scheduling escalation to SIGKILL in 90secs from now
I0823 16:32:50.426288 21665 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:50.430682 21682 default_executor.cpp:202] Received ACKNOWLEDGED event
W0823 16:32:50.917691 21689 health_checker.cpp:273] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed: Command timed out after 5secs
W0823 16:32:50.917750 21689 health_checker.cpp:305] COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' failed 5 times consecutively
I0823 16:32:50.917850 21680 default_executor.cpp:1375] Received task health update for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka', task is not healthy
I0823 16:32:50.919066 21680 default_executor.cpp:1249] Received kill for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka'
I0823 16:32:50.919119 21680 checker_process.cpp:281] Stopped COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka'
I0823 16:32:50.919231 21680 default_executor.cpp:1124] Killing task test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.8e04f74f-cb8b-46b9-8758-340455a844c8 with SIGTERM signal
I0823 16:32:50.919241 21680 default_executor.cpp:1135] Scheduling escalation to SIGKILL in 90secs from now
I0823 16:32:51.127272 21651 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:51.130367 21670 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:51.973668 21665 default_executor.cpp:953] Child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.8e04f74f-cb8b-46b9-8758-340455a844c8 of task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka' completed in state TASK_KILLED: Command terminated with signal Terminated
I0823 16:32:51.973721 21665 default_executor.cpp:974] Killing task group containing tasks [ test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.akka, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis, test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery ]
I0823 16:32:51.973819 21691 checker_process.cpp:281] Stopped COMMAND health check for task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis'
I0823 16:32:51.973997 21665 default_executor.cpp:1124] Killing task test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 with SIGTERM signal
I0823 16:32:51.974021 21665 default_executor.cpp:1135] Scheduling escalation to SIGKILL in 3secs from now
I0823 16:32:51.975106 21665 default_executor.cpp:953] Child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.ab481072-c8ab-4a76-be8b-7f4431220e7b of task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.delivery' completed in state TASK_KILLED: Command terminated with signal Terminated
I0823 16:32:51.995775 21671 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:51.997719 21644 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:52.003360 21676 default_executor.cpp:202] Received ACKNOWLEDGED event
I0823 16:32:54.974514 21646 default_executor.cpp:1213] Task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 did not terminate after 3secs, sending SIGKILL to the container
W0823 16:32:54.982900 21650 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 failed: The agent failed to send signal Killed (9) to the container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7: Unable to send signal to container: No such process; Retrying in 1secs
I0823 16:32:55.983327 21639 default_executor.cpp:1213] Task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 did not terminate after 3secs, sending SIGKILL to the container
W0823 16:32:55.990069 21644 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 failed: The agent failed to send signal Killed (9) to the container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7: Unable to send signal to container: No such process; Retrying in 1secs
I0823 16:32:56.991422 21670 default_executor.cpp:1213] Task 'test_cb88dd0c-a6e0-11e8-888f-fb74b926ae8c.instance-08d37bd7-a6e1-11e8-9e12-0242e3789894.redis' running in child container 3680beff-96d2-4ebd-832c-9cbbddf8c507.fc60bf0f-5814-4ea9-a37f-89ebe3e2f5f7 did not terminate after 3secs, sending SIGKILL to the container

      and then it loops forever with retrying to kill already non-existing process

      the other form of that bug we observed is

      I0823 11:19:44.460397 35632 default_executor.cpp:202] Received KILL event
I0823 11:19:44.460433 35632 default_executor.cpp:1249] Received kill for task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis'
W0823 11:19:44.460445 35632 default_executor.cpp:1259] Ignoring kill for task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' as it is in the process of getting killed
I0823 11:19:45.078868 35660 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:45.083555 35645 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:46.084547 35637 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:46.088583 35639 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:47.089757 35630 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:47.094741 35631 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:48.095813 35623 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:48.100821 35632 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:49.101478 35627 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:49.105983 35651 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:50.106503 35662 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:50.111423 35723 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:51.112059 35725 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:51.116915 35664 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:52.118046 35653 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:52.122288 35616 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:53.123337 35658 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:53.128535 35641 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:54.129462 35633 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:54.133767 35644 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:55.134635 35618 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:55.138553 35622 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:56.139037 35638 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:56.142948 35724 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:57.143637 35659 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:57.148473 35636 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:58.149035 35648 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:58.152792 35621 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:19:59.153236 35629 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:19:59.157325 35656 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:00.158377 35660 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:20:00.162392 35627 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:01.162860 35637 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:20:01.167155 35662 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:02.167553 35630 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:20:02.172479 35725 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:03.173439 35619 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:20:03.177597 35653 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:04.178180 35727 default_executor.cpp:1213] Task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 did not terminate after 1mins, sending SIGKILL to the container
W0823 11:20:04.182360 35658 default_executor.cpp:1222] Escalation to SIGKILL the task 'test_11c6bfe0-a660-11e8-8861-4f65393a63f6.instance-687724c4-a660-11e8-ab64-c6905d8f8b70.redis' running in child container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53 failed: The agent failed to send signal Killed (9) to the container ab8332fe-bd03-47b0-962d-cc1d724a9f13.aa1ffbe8-816c-400b-ad6d-0413b0c1ec53: Unable to send signal to container: No such process; Retrying in 1secs
I0823 11:20:04.460697 35662 default_executor.cpp:202] Received KILL event

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. MESOS-8679 If the first KILL stuck in the default executor, all other KILLs will be ignored.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              Kirill P Kirill Plyashkevich
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: