[MESOS-4912] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers fails.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.28.0
• Fix Version/s: 1.0.0
• Component/s: containerization
• Labels: mesosphere

Environment

CenOS 7, SSL

• Epic Link: flaky
• Sprint: Mesosphere Sprint 31
• Story Points: 3
• Assignee: Gilbert Song
• Reporter: Bernd Mathiske
• Shepherd: Jie Yu
• Created: 10/Mar/16 15:37
• Updated: 26/Nov/18 12:20
• Resolved: 26/Nov/18 12:20

Description

Observed on our CI:

[09:34:15] :	 [Step 11/11] [ RUN      ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
[09:34:19]W:	 [Step 11/11] I0309 09:34:19.906719  2357 linux.cpp:81] Making '/tmp/MLVLnv' a shared mount
[09:34:19]W:	 [Step 11/11] I0309 09:34:19.923548  2357 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[09:34:19]W:	 [Step 11/11] I0309 09:34:19.924705  2376 containerizer.cpp:666] Starting container 'da610f7f-a709-4de8-94d3-74f4a520619b' for executor 'test_executor1' of framework ''
[09:34:19]W:	 [Step 11/11] I0309 09:34:19.925355  2371 provisioner.cpp:285] Provisioning image rootfs '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0' for container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:19]W:	 [Step 11/11] I0309 09:34:19.925881  2377 copy.cpp:127] Copying layer path '/tmp/MLVLnv/test_image1' to rootfs '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0'
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.835127  2376 linux.cpp:355] Bind mounting work directory from '/tmp/MLVLnv/slaves/test_slave/frameworks/executors/test_executor1/runs/da610f7f-a709-4de8-94d3-74f4a520619b' to '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0/mnt/mesos/sandbox' for container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.835392  2376 linux.cpp:683] Changing the ownership of the persistent volume at '/tmp/MLVLnv/volumes/roles/test_role/persistent_volume_id' with uid 0 and gid 0
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.840425  2376 linux.cpp:723] Mounting '/tmp/MLVLnv/volumes/roles/test_role/persistent_volume_id' to '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0/mnt/mesos/sandbox/volume' for persistent volume disk(test_role)[persistent_volume_id:volume]:32 of container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.843878  2374 linux_launcher.cpp:304] Cloning child process with flags = CLONE_NEWNS
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.848302  2371 containerizer.cpp:666] Starting container 'fe4729c5-1e63-4cc6-a2e3-fe5006ffe087' for executor 'test_executor2' of framework ''
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.848758  2371 containerizer.cpp:1392] Destroying container 'da610f7f-a709-4de8-94d3-74f4a520619b'
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.848865  2373 provisioner.cpp:285] Provisioning image rootfs '/tmp/MLVLnv/provisioner/containers/fe4729c5-1e63-4cc6-a2e3-fe5006ffe087/backends/copy/rootfses/518b2464-43dd-47b0-9648-e78aedde6917' for container fe4729c5-1e63-4cc6-a2e3-fe5006ffe087
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.849449  2375 copy.cpp:127] Copying layer path '/tmp/MLVLnv/test_image2' to rootfs '/tmp/MLVLnv/provisioner/containers/fe4729c5-1e63-4cc6-a2e3-fe5006ffe087/backends/copy/rootfses/518b2464-43dd-47b0-9648-e78aedde6917'
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.854038  2374 cgroups.cpp:2427] Freezing cgroup /sys/fs/cgroup/freezer/mesos/da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.856693  2372 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/da610f7f-a709-4de8-94d3-74f4a520619b after 2.608128ms
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.859237  2377 cgroups.cpp:2445] Thawing cgroup /sys/fs/cgroup/freezer/mesos/da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.861454  2377 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/da610f7f-a709-4de8-94d3-74f4a520619b after 2176us
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.934608  2378 containerizer.cpp:1608] Executor for container 'da610f7f-a709-4de8-94d3-74f4a520619b' has exited
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.937692  2372 linux.cpp:798] Unmounting volume '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0/mnt/mesos/sandbox/volume' for container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.937742  2372 linux.cpp:817] Unmounting sandbox/work directory '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0/mnt/mesos/sandbox' for container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:30]W:	 [Step 11/11] I0309 09:34:30.938129  2375 provisioner.cpp:330] Destroying container rootfs at '/tmp/MLVLnv/provisioner/containers/da610f7f-a709-4de8-94d3-74f4a520619b/backends/copy/rootfses/0d7e047a-50f1-490b-bb58-00e9c49628d0' for container da610f7f-a709-4de8-94d3-74f4a520619b
[09:34:45] :	 [Step 11/11] ../../src/tests/containerizer/filesystem_isolator_tests.cpp:1318: Failure
[09:34:45] :	 [Step 11/11] Failed to wait 15secs for wait1
[09:34:48] :	 [Step 11/11] [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers (32341 ms)

Issue Links

• relates to: MESOS-5192 LinuxFilesystemIsolatorTest.ROOT_MultipleContainers is flaky