[MESOS-5175] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint is flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: flaky, test
• Labels: filesystem isolation mesosphere

Environment

CentOS 7 with SSL and libevent enabled

• Epic Link: flaky
• Sprint: Mesosphere Sprint 37
• Assignee: Gilbert Song
• Reporter: Greg Mann
• Created: 11/Apr/16 20:10
• Updated: 26/Apr/17 17:01
• Resolved: 23/Jun/16 16:04

Description

Observed on the internal Mesosphere CI:

[07:12:07] :	 [Step 11/11] [ RUN      ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint
[07:12:08]W:	 [Step 11/11] I0410 07:12:08.906998 32129 linux.cpp:81] Making '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH' a shared mount
[07:12:08]W:	 [Step 11/11] I0410 07:12:08.923028 32129 linux_launcher.cpp:101] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
[07:12:08]W:	 [Step 11/11] I0410 07:12:08.923751 32144 containerizer.cpp:682] Starting container '86d04a91-e7b0-4b8f-9706-b9969796b5d1' for executor 'test_executor' of framework ''
[07:12:08]W:	 [Step 11/11] I0410 07:12:08.924296 32148 provisioner.cpp:285] Provisioning image rootfs '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab' for container 86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:08]W:	 [Step 11/11] I0410 07:12:08.924885 32145 copy.cpp:127] Copying layer path '/tmp/WwQa3Q/test_image' to rootfs '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab'
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.627612 32145 linux.cpp:355] Bind mounting work directory from '/tmp/WwQa3Q/sandbox' to '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab/mnt/mesos/sandbox' for container 86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.648669 32147 linux_launcher.cpp:281] Cloning child process with flags = CLONE_NEWNS
[07:12:13]W:	 [Step 11/11] + /mnt/teamcity/work/4240ba9ddd0997c3/build/src/mesos-containerizer mount --help=false --operation=make-rslave --path=/
[07:12:13]W:	 [Step 11/11] + grep -E /mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/.+ /proc/self/mountinfo
[07:12:13]W:	 [Step 11/11] + grep -v 86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:13]W:	 [Step 11/11] + cut '-d ' -f5
[07:12:13]W:	 [Step 11/11] + xargs --no-run-if-empty umount -l
[07:12:13]W:	 [Step 11/11] + mount -n --rbind /tmp/WwQa3Q /mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab/mnt/mesos/sandbox/mountpoint
[07:12:13] :	 [Step 11/11] Changing root to /mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.827551 32145 containerizer.cpp:1674] Executor for container '86d04a91-e7b0-4b8f-9706-b9969796b5d1' has exited
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.827607 32145 containerizer.cpp:1439] Destroying container '86d04a91-e7b0-4b8f-9706-b9969796b5d1'
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.830469 32145 cgroups.cpp:2676] Freezing cgroup /sys/fs/cgroup/freezer/mesos/86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.832928 32143 cgroups.cpp:1409] Successfully froze cgroup /sys/fs/cgroup/freezer/mesos/86d04a91-e7b0-4b8f-9706-b9969796b5d1 after 2.412032ms
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.835292 32150 cgroups.cpp:2694] Thawing cgroup /sys/fs/cgroup/freezer/mesos/86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.837411 32150 cgroups.cpp:1438] Successfullly thawed cgroup /sys/fs/cgroup/freezer/mesos/86d04a91-e7b0-4b8f-9706-b9969796b5d1 after 2.07616ms
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.840045 32148 linux.cpp:817] Unmounting sandbox/work directory '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab/mnt/mesos/sandbox' for container 86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:13]W:	 [Step 11/11] I0410 07:12:13.840504 32150 provisioner.cpp:330] Destroying container rootfs at '/mnt/teamcity/temp/buildTmp/LinuxFilesystemIsolatorTest_ROOT_VolumeFromHostSandboxMountPoint_aSovaH/provisioner/containers/86d04a91-e7b0-4b8f-9706-b9969796b5d1/backends/copy/rootfses/104f1991-f54a-4dd0-ab92-48ff2d3bebab' for container 86d04a91-e7b0-4b8f-9706-b9969796b5d1
[07:12:28] :	 [Step 11/11] ../../src/tests/containerizer/filesystem_isolator_tests.cpp:961: Failure
[07:12:28] :	 [Step 11/11] Failed to wait 15secs for wait
[07:12:30] :	 [Step 11/11] [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint (23703 ms)

Issue Links

• is related to: MESOS-4943 Reduce the size of LinuxRootfs in tests.