[MESOS-5192] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Labels: flaky flaky-test isolation mesosphere

Environment

CentOS 7 + SSL

• Epic Link: flaky
• Sprint: Mesosphere Sprint 37
• Assignee: Gilbert Song
• Reporter: Neil Conway
• Created: 12/Apr/16 14:24
• Updated: 23/Jun/16 15:51
• Resolved: 23/Jun/16 15:51

Description

Observed on internal Mesosphere CI:

[11:32:03] :	 [Step 11/11] [ RUN      ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers
[11:32:09]W:	 [Step 11/11] I0412 11:32:09.587877 17436 linux.cpp:81] Making '/tmp/NMWl31' a shared mount
[11:32:09]W:	 [Step 11/11] I0412 11:32:09.603153 17436 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[11:32:09]W:	 [Step 11/11] I0412 11:32:09.604372 17456 containerizer.cpp:682] Starting container 'f1f5de4c-aaef-45c7-b28c-83014327eb40' for executor 'test_executor1' of framework ''
[11:32:09]W:	 [Step 11/11] I0412 11:32:09.604940 17450 provisioner.cpp:285] Provisioning image rootfs '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53' for container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:09]W:	 [Step 11/11] I0412 11:32:09.605561 17454 copy.cpp:128] Copying layer path '/tmp/NMWl31/test_image1' to rootfs '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53'
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.784813 17450 linux.cpp:355] Bind mounting work directory from '/tmp/NMWl31/slaves/test_slave/frameworks/executors/test_executor1/runs/f1f5de4c-aaef-45c7-b28c-83014327eb40' to '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53/mnt/mesos/sandbox' for container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.785079 17450 linux.cpp:683] Changing the ownership of the persistent volume at '/tmp/NMWl31/volumes/roles/test_role/persistent_volume_id' with uid 0 and gid 0
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.790043 17450 linux.cpp:723] Mounting '/tmp/NMWl31/volumes/roles/test_role/persistent_volume_id' to '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53/mnt/mesos/sandbox/volume' for persistent volume disk(test_role)[persistent_volume_id:volume]:32 of container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.791724 17455 linux_launcher.cpp:281] Cloning child process with flags = CLONE_NEWNS
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.797466 17453 containerizer.cpp:682] Starting container '2c9bd571-7804-481d-9a42-1df65489dda8' for executor 'test_executor2' of framework ''
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.797947 17453 containerizer.cpp:1439] Destroying container 'f1f5de4c-aaef-45c7-b28c-83014327eb40'
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.798017 17457 provisioner.cpp:285] Provisioning image rootfs '/tmp/NMWl31/provisioner/containers/2c9bd571-7804-481d-9a42-1df65489dda8/backends/copy/rootfses/3da54ab5-8871-4913-a9d1-ae658286023a' for container 2c9bd571-7804-481d-9a42-1df65489dda8
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.798683 17451 copy.cpp:128] Copying layer path '/tmp/NMWl31/test_image2' to rootfs '/tmp/NMWl31/provisioner/containers/2c9bd571-7804-481d-9a42-1df65489dda8/backends/copy/rootfses/3da54ab5-8871-4913-a9d1-ae658286023a'
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.802534 17456 cgroups.cpp:2676] Freezing cgroup /sys/fs/cgroup/freezer/mesos/f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.804713 17451 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/f1f5de4c-aaef-45c7-b28c-83014327eb40 after 2.123008ms
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.807065 17450 cgroups.cpp:2694] Thawing cgroup /sys/fs/cgroup/freezer/mesos/f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.809074 17450 cgroups.cpp:1438] Successfully thawed cgroup /sys/fs/cgroup/freezer/mesos/f1f5de4c-aaef-45c7-b28c-83014327eb40 after 1.96992ms
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.984756 17457 containerizer.cpp:1674] Executor for container 'f1f5de4c-aaef-45c7-b28c-83014327eb40' has exited
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.987437 17451 linux.cpp:798] Unmounting volume '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53/mnt/mesos/sandbox/volume' for container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.987499 17451 linux.cpp:817] Unmounting sandbox/work directory '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53/mnt/mesos/sandbox' for container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:32:13]W:	 [Step 11/11] I0412 11:32:13.987915 17455 provisioner.cpp:338] Destroying container rootfs at '/tmp/NMWl31/provisioner/containers/f1f5de4c-aaef-45c7-b28c-83014327eb40/backends/copy/rootfses/1dcaba0b-23ab-462f-be91-cd49090c4f53' for container f1f5de4c-aaef-45c7-b28c-83014327eb40
[11:33:13] :	 [Step 11/11] ../../src/tests/containerizer/filesystem_isolator_tests.cpp:1324: Failure
[11:33:13] :	 [Step 11/11] Failed to wait 1mins for wait1
[11:33:19] :	 [Step 11/11] [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_MultipleContainers (76539 ms)

Issue Links

• is blocked by: MESOS-4943 Reduce the size of LinuxRootfs in tests.
• is related to: MESOS-4912 LinuxFilesystemIsolatorTest.ROOT_MultipleContainers fails.