[MESOS-9268] Hitting agent's `/containers` endpoint might backlog Docker containerizer process.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: agent, containerization, docker
• Assignee: Unassigned
• Reporter: Andrei Budnik
• Created: 26/Sep/18 17:05
• Updated: 21/Dec/18 17:39
• Resolved: 18/Oct/18 17:13

Description

When the agent's `/containers` endpoint is hit, the agent calls `DockerContainerizerProcess::usage()` method for every running Docker container. If there are lots (hundreds) of running Docker containers and the system is under heavy load, then `DockerContainerizerProcess::usage()` method takes a lot of time to be processed. Hence, when the Docker containerizer's mailbox is full of `DockerContainerizerProcess::usage()` commands waiting to be processed, an attempt to launch a new Docker container will lead to putting `DockerContainerizer::launch()` into a long queue. This is an example from logs:

2018-09-07 00:06:03: I0907 00:06:03.031744 65329 slave.cpp:2857] Launching container 320547f2-9ab9-4ade-bddb-bfd6f5ed0834 for executor 'example_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1' of framework 51b197db-56ae-47dc-a1bd-3aaf2306bd1a-0001
...
2018-09-07 00:36:20: I0907 00:36:20.472048 65307 docker.cpp:1179] Starting container '320547f2-9ab9-4ade-bddb-bfd6f5ed0834' for task 'example_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1' (and executor 'middleware_test2519.ceeefb8a-b231-11e8-8d62-8e820bbfbfd1') of framework 51b197db-56ae-47dc-a1bd-3aaf2306bd1a-0001

After disabling `/containers` endpoint, the issue with a backlogged Docker containerizer disappears.

Issue Links

• is related to: MESOS-9279 Docker Containerizer 'usage' call might be expensive if mount table is big.
• is related to: MESOS-9283 Docker containerizer actor can get backlogged with large number of containers.