Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-9142

CNI detach might fail due to missing network config file.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.1, 1.5.1, 1.6.1, 1.7.0
    • 1.5.2, 1.6.2, 1.7.0
    • None
    • None

    Description

      Observed this in one of the scale testing:

      If the container is destroyed while in PREPARING state, the network config file won't be there. The CNI detach in that case should return success because attach is not called yet.

      Aug 08 23:52:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:52:03.762208  4045 slave.cpp:3562] Launching container 278d88a8-2bd1-4359-8c70-25dc90879cdc for executor 'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework 4397555f-b8fd-478c-907a-7532e2f5e32e-0001
Aug 08 23:53:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:53:03.232110  4047 containerizer.cpp:1211] Starting container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:53:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:53:03.249105  4049 provisioner.cpp:546] Provisioning image rootfs '/var/lib/mesos/slave/provisioner/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc/backends/overlay/rootfses/25dc9ff3-a4ad-4b59-bdf7-02f7e741c729' for container 278d88a8-2bd1-4359-8c70-25dc90879cdc using overlay backend
Aug 08 23:53:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:53:31.367316  4047 containerizer.cpp:2981] Transitioning the state of container 278d88a8-2bd1-4359-8c70-25dc90879cdc from PROVISIONING to PREPARING
Aug 08 23:55:22 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:55:22.988091  4047 memory.cpp:479] Started listening for OOM events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:55:23.196844  4047 memory.cpp:591] Started listening on 'low' memory pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:55:23.277029  4047 memory.cpp:591] Started listening on 'medium' memory pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:55:23 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:55:23.407688  4047 memory.cpp:591] Started listening on 'critical' memory pressure events for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:57:31.760859  4043 memory.cpp:199] Updated 'memory.soft_limit_in_bytes' to 64MB for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:31 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:57:31.878435  4043 memory.cpp:228] Updated 'memory.limit_in_bytes' to 64MB for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:35 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:57:35.892071  4044 cpu.cpp:101] Updated 'cpu.shares' to 112 (cpus 0.11) for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 08 23:57:36 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0808 23:57:36.132098  4044 cpu.cpp:121] Updated 'cpu.cfs_period_us' to 100ms and 'cpu.cfs_quota_us' to 11ms (cpus 0.11) for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:01.791821  4047 isolator_module.cpp:68] Container prepare: container_id[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"] container_config[directory: "/var/lib/mesos/slave/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc" user: "root" rootfs: "/var/lib/mesos/slave/provisioner/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc/backends/overlay/rootfses/25dc9ff3-a4ad-4b59-bdf7-02f7e741c729" docker { manifest { id: "d7066268059eb2499922833bd058e9e0b9da8365bcae4b85ee22296b02fa0489" parent: "f18ee96f0b1656cab52554b270f19e8df5046d307296d2146539c04565d67747" created: "2018-07-06T14:14:06.393355914Z" container: "b449414436d7bc63cc85dcd24abde14bcec1d92f54f2316212059ee2ed7f3a65" container_config { Hostname: "b449414436d7" Env: "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" User: "" Cmd: "/bin/sh" Cmd: "-c" Cmd: "#(nop) " Cmd: "CMD [\"/bin/sh\"]" WorkingDir: "" Image: "sha256:9025f4d6338c3425933e4734e8a27fb615d56ca8862e78bb8c4e2426f2db78bd" } docker_version: "17.06.2-ce" config { Hostname: "" Env: "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" User: "" Cmd: "/bin/sh" WorkingDir: "" Image: "sha256:9025f4d6338c3425933e4734e8a27fb615d56ca8862e78bb8c4e2426f2db78bd" } architecture: "amd64" os: "linux" } } executor_info { executor_id { value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } resources { name: "cpus" type: SCALAR scalar { value: 0.11 } allocation_info { role: "slave_public" } } resources { name: "mem" type: SCALAR scalar { value: 64 } allocation_info { role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range { begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } } command { environment { variables { name: "MARATHON_APP_VERSION" value: "2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" } variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables { name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name: "MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } variables { name: "PORT" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" } variables { name: "PORT_80" value: "19471" } variables { name: "MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value: "/testcni2" } variables { name: "PORT0" value: "19471" } } value: "/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor" shell: false arguments: "mesos-executor" arguments: "--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments: "--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" } framework_id { value: "4397555f-b8fd-478c-907a-7532e2f5e32e-0001" } name: "Command Executor (Task: testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a) (Command: sh -c \'sleep 10000\')" source: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" container { type: MESOS mesos { image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos { labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80 protocol: "tcp" } } } discovery { visibility: FRAMEWORK name: "testcni2" ports { ports { number: 19471 protocol: "tcp" labels { labels { key: "network-scope" value: "host" } } } } } labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } } task_info { name: "testcni2" task_id { value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } slave_id { value: "4397555f-b8fd-478c-907a-7532e2f5e32e-S0" } resources { name: "cpus" type: SCALAR scalar { value: 0.01 } allocation_info { role: "slave_public" } } resources { name: "mem" type: SCALAR scalar { value: 32 } allocation_info { role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range { begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } } command { environment { variables { name: "MARATHON_APP_VERSION" value: "2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" } variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables { name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name: "MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } variables { name: "PORT" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" } variables { name: "PORT_80" value: "19471" } variables { name: "MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value: "/testcni2" } variables { name: "PORT0" value: "19471" } } value: "sleep 10000" } container { type: MESOS mesos { image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos { labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80 protocol: "tcp" } } } labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } discovery { visibility: FRAMEWORK name: "testcni2" ports { ports { number: 19471 protocol: "tcp" labels { labels { key: "network-scope" value: "host" } } } } } } command_info { environment { variables { name: "MARATHON_APP_VERSION" value: "2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" } variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables { name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name: "MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } variables { name: "PORT" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" } variables { name: "PORT_80" value: "19471" } variables { name: "MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value: "/testcni2" } variables { name: "PORT0" value: "19471" } } value: "/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor" shell: false arguments: "mesos-executor" arguments: "--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments: "--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" } container_info { type: MESOS mesos { image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos { labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80 protocol: "tcp" } } } resources { name: "cpus" type: SCALAR scalar { value: 0.11 } allocation_info { role: "slave_public" } } resources { name: "mem" type: SCALAR scalar { value: 64 } allocation_info { role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range { begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } }]
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:01.792201  4047 container_assigner.cpp:64] Registering and retrieving endpoint for container_id[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"] executor_info[executor_id { value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } resources { name: "cpus" type: SCALAR scalar { value: 0.11 } allocation_info { role: "slave_public" } } resources { name: "mem" type: SCALAR scalar { value: 64 } allocation_info { role: "slave_public" } } resources { name: "ports" type: RANGES ranges { range { begin: 19471 end: 19471 } } allocation_info { role: "slave_public" } } command { environment { variables { name: "MARATHON_APP_VERSION" value: "2018-08-08T23:45:27.052Z" } variables { name: "HOST" value: "10.0.3.181" } variables { name: "MARATHON_APP_RESOURCE_CPUS" value: "0.01" } variables { name: "MARATHON_APP_RESOURCE_GPUS" value: "0" } variables { name: "MESOS_TASK_ID" value: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" } variables { name: "PORT" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_MEM" value: "32.0" } variables { name: "PORTS" value: "19471" } variables { name: "MARATHON_APP_RESOURCE_DISK" value: "0.0" } variables { name: "PORT_80" value: "19471" } variables { name: "MARATHON_APP_LABELS" value: "" } variables { name: "MARATHON_APP_ID" value: "/testcni2" } variables { name: "PORT0" value: "19471" } } value: "/opt/mesosphere/packages/mesos--39041edf914496bb882949ade1e78043d53dadd0/libexec/mesos/mesos-executor" shell: false arguments: "mesos-executor" arguments: "--launcher_dir=/opt/mesosphere/active/mesos/libexec/mesos" arguments: "--sandbox_directory=/mnt/mesos/sandbox" arguments: "--user=root" } framework_id { value: "4397555f-b8fd-478c-907a-7532e2f5e32e-0001" } name: "Command Executor (Task: testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a) (Command: sh -c \'sleep 10000\')" source: "testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a" container { type: MESOS mesos { image { type: DOCKER docker { name: "alpine" } cached: true } } network_infos { labels { labels { key: "DCOS_SPACE" value: "/testcni2" } } ip_addresses { } name: "mesos-bridge" port_mappings { host_port: 19471 container_port: 80 protocol: "tcp" } } } discovery { visibility: FRAMEWORK name: "testcni2" ports { ports { number: 19471 protocol: "tcp" labels { labels { key: "network-scope" value: "host" } } } } } labels { labels { key: "DCOS_SPACE" value: "/testcni2" } }].
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:01.822054  4195 container_assigner_strategy.cpp:156] New ephemeral-port reader for container[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"] created at endpoint[198.51.100.1:47454].
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:01.822118  4195 container_state_cache_impl.cpp:134] Writing container file[/run/dcos/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc] with endpoint[198.51.100.1:47454]
Aug 09 00:02:01 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:01.828858  4045 secret.cpp:296] 0 secrets have been resolved for container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:02:03 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:03.230397  4047 switchboard.cpp:317] Container logger module finished preparing container 278d88a8-2bd1-4359-8c70-25dc90879cdc; IOSwitchboard server is not required
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:04.510732  4045 containerizer.cpp:2367] Destroying container 278d88a8-2bd1-4359-8c70-25dc90879cdc in PREPARING state
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:04.511010  4045 containerizer.cpp:2981] Transitioning the state of container 278d88a8-2bd1-4359-8c70-25dc90879cdc from PREPARING to DESTROYING
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:04.745018  4042 container_assigner.cpp:101] Unregistering container_id[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"].
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:04.746515  4195 container_assigner_strategy.cpp:202] Closing ephemeral-port reader for container[value: "278d88a8-2bd1-4359-8c70-25dc90879cdc"] at endpoint[198.51.100.1:47454].
Aug 09 00:02:04 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:02:04.749131  4195 container_state_cache_impl.cpp:145] Removing container file[/run/dcos/mesos/isolators/com_mesosphere_MetricsIsolatorModule/containers/278d88a8-2bd1-4359-8c70-25dc90879cdc]
Aug 09 00:02:05 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: E0809 00:02:05.153556  4048 slave.cpp:6201] Container '278d88a8-2bd1-4359-8c70-25dc90879cdc' for executor 'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework 4397555f-b8fd-478c-907a-7532e2f5e32e-0001 failed to start: Container is being destroyed during preparing
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: E0809 00:07:08.151012  4045 slave.cpp:6308] Termination of executor 'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' of framework 4397555f-b8fd-478c-907a-7532e2f5e32e-0001 failed: Failed to clean up an isolator when destroying container: Failed to parse CNI network configuration file: '/run/mesos/isolators/network/cni/278d88a8-2bd1-4359-8c70-25dc90879cdc/mesos-bridge/network.conf': Failed to read CNI network configuration file: '/run/mesos/isolators/network/cni/278d88a8-2bd1-4359-8c70-25dc90879cdc/mesos-bridge/network.conf': Failed to open file: No such file or directory
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: E0809 00:07:08.151798  4046 slave.cpp:5639] Failed to update resources for container 278d88a8-2bd1-4359-8c70-25dc90879cdc of executor 'testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a' running task testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a on status update for terminal task, destroying container: Container not found
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:08.152240  4043 composing.cpp:600] Attempted to destroy unknown container 278d88a8-2bd1-4359-8c70-25dc90879cdc
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:07:08.164916  4047 gc.cpp:95] Scheduling '/var/lib/mesos/slave/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc' for gc 1.99999809405037days in the future
Aug 09 00:07:08 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: I0809 00:07:08.165079  4047 gc.cpp:95] Scheduling '/var/lib/mesos/slave/meta/slaves/4397555f-b8fd-478c-907a-7532e2f5e32e-S0/frameworks/4397555f-b8fd-478c-907a-7532e2f5e32e-0001/executors/testcni2.09d9faa5-9b66-11e8-b04f-e2837c6f5f2a/runs/278d88a8-2bd1-4359-8c70-25dc90879cdc' for gc 1.99999809211852days in the future
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:26.571424  4047 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for subsystem 'memory': Unknown container
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:26.571480  4047 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read 'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:26 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:26.571648  4047 containerizer.cpp:2221] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:27.801781  4043 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read from 'blkio.time': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:27.801839  4043 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for subsystem 'memory': Unknown container
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:27.801862  4043 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read 'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:27 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:27.801961  4043 containerizer.cpp:2221] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:29.182669  4048 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read from 'blkio.time': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:29.182729  4048 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to get usage for subsystem 'memory': Unknown container
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:29.182752  4048 cgroups.cpp:737] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Failed to read 'cpu.stat': 'mesos/278d88a8-2bd1-4359-8c70-25dc90879cdc' is not a valid cgroup
Aug 09 00:07:29 ip-10-0-3-181.us-west-2.compute.internal mesos-agent[4022]: W0809 00:07:29.182870  4046 containerizer.cpp:2221] Skipping resource statistic for container 278d88a8-2bd1-4359-8c70-25dc90879cdc because: Unknown container

      Attachments

        Activity

          People

            jieyu Jie Yu
            jieyu Jie Yu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: