[MESOS-8877] Docker container's resources will be wrongly enlarged in cgroups after agent recovery

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.9.0
• Fix Version/s: 1.10.0
• Component/s: docker
• Labels: containerization
• Sprint: Studio 1: RI-23 68
• Story Points: 3
• Assignee: Qian Zhang
• Reporter: Qian Zhang
• Created: 03/May/18 09:25
• Updated: 05/May/20 08:26
• Resolved: 05/May/20 08:26

Description

Reproduce steps:

1. Run `mesos-execute --master=10.0.49.2:5050 --task=file:///home/qzhang/workspace/config/task_docker.json --checkpoint=true` to launch a Docker container.

# cat task_docker.json 
{
  "name": "test",
  "task_id": {"value" : "test"},
  "agent_id": {"value" : ""},
  "resources": [
    {"name": "cpus", "type": "SCALAR", "scalar": {"value": 0.1}},
    {"name": "mem", "type": "SCALAR", "scalar": {"value": 32}}
  ],
  "command": {
    "value": "sleep 55555"
  },
  "container": {
    "type": "DOCKER",
    "docker": {
      "image": "alpine"
    }
  }
}

2. When the Docker container is running, we can see its resources in cgroups are correctly set, so far so good.

# cat /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/cpu.cfs_quota_us 
10000
# cat /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/memory.limit_in_bytes 
33554432

3. Restart Mesos agent, and then we will see the resources of the Docker container will be wrongly enlarged.

I0503 02:06:17.268340 29512 docker.cpp:1855] Updated 'cpu.shares' to 204 at /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106 for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88
I0503 02:06:17.271390 29512 docker.cpp:1882] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 20ms (cpus 0.2) for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88
I0503 02:06:17.273082 29512 docker.cpp:1924] Updated 'memory.soft_limit_in_bytes' to 64MB for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88
I0503 02:06:17.275908 29512 docker.cpp:1950] Updated 'memory.limit_in_bytes' to 64MB at /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106 for container 1b21295b-2f49-4d08-84c7-43b9ae15ad88

# cat /sys/fs/cgroup/cpu,cpuacct/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/cpu.cfs_quota_us
20000
# cat /sys/fs/cgroup/memory/docker/a711b3c7b0d91cd6d1c7d8daf45a90ff78d2fd66973e615faca55a717ec6b106/memory.limit_in_bytes
67108864

Issue Links

• is related to: MESOS-8840 `cpu.cfs_quota_us` may be accidentally set for command task using docker during agent recovery.