[MESOS-3050] Failing ROOT_ tests on CentOS 7.1

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 0.23.0, 0.24.0
• Component/s: containerization, docker, test
• Labels: mesosphere

Environment

CentOS Linux release 7.1.1503
0.24.0

• Sprint: Mesosphere Sprint 16
• Story Points: 5
• Assignee: Marco Massenzio
• Reporter: Adam B
• Shepherd: Vinod Kone
• Created: 15/Jul/15 00:50
• Updated: 20/Aug/15 17:59
• Resolved: 20/Aug/15 17:59

Description

Running `sudo make check` on CentOS 7.1 for Mesos 0.23.0-rc3 causes several several failures/errors:

[ RUN      ] DockerTest.ROOT_DOCKER_CheckPortResource
../../src/tests/docker_tests.cpp:303: Failure
(run).failure(): Container exited on error: exited with status 1
[  FAILED  ] DockerTest.ROOT_DOCKER_CheckPortResource (709 ms)

...

[ RUN      ] PerfEventIsolatorTest.ROOT_CGROUPS_Sample
../../src/tests/isolator_tests.cpp:837: Failure
isolator: Failed to create PerfEvent isolator, invalid events: { cycles, task-clock }
[  FAILED  ] PerfEventIsolatorTest.ROOT_CGROUPS_Sample (9 ms)
[----------] 1 test from PerfEventIsolatorTest (9 ms total)
   
[----------] 2 tests from SharedFilesystemIsolatorTest
[ RUN      ] SharedFilesystemIsolatorTest.ROOT_RelativeVolume
+ mount -n --bind /tmp/SharedFilesystemIsolatorTest_ROOT_RelativeVolume_4yTEAC/var/tmp /var/tmp
+ touch /var/tmp/492407e1-5dec-4b34-8f2f-130430f41aac
../../src/tests/isolator_tests.cpp:1001: Failure
Value of: os::exists(file)
  Actual: true
Expected: false
[  FAILED  ] SharedFilesystemIsolatorTest.ROOT_RelativeVolume (92 ms)
[ RUN      ] SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume
+ mount -n --bind /tmp/SharedFilesystemIsolatorTest_ROOT_AbsoluteVolume_OwYrXK /var/tmp
+ touch /var/tmp/7de712aa-52eb-4976-b0f9-32b6a006418d
../../src/tests/isolator_tests.cpp:1086: Failure
Value of: os::exists(path::join(containerPath, filename))
  Actual: true
Expected: false
[  FAILED  ] SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume (100 ms)

...

[----------] 1 test from UserCgroupIsolatorTest/0, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup
-bash: /sys/fs/cgroup/blkio/user.slice/cgroup.procs: Permission denied
mkdir: cannot create directory ‘/sys/fs/cgroup/blkio/user.slice/user’: Permission denied
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/blkio/user.slice/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/memory/mesos/bbf8c8f0-3d67-40df-a269-b3dc6a9597aa/cgroup.procs: Permission denied
-bash: /sys/fs/cgroup/cpuacct,cpu/user.slice/cgroup.procs: No such file or directory
mkdir: cannot create directory ‘/sys/fs/cgroup/cpuacct,cpu/user.slice/user’: No such file or directory
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/cpuacct,cpu/user.slice/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/cgroup.procs: No such file or directory
mkdir: cannot create directory ‘/sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user’: No such file or directory
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
[  FAILED  ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess (1034 ms)
[----------] 1 test from UserCgroupIsolatorTest/0 (1034 ms total)
[----------] 1 test from UserCgroupIsolatorTest/1, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup
-bash: /sys/fs/cgroup/blkio/user.slice/cgroup.procs: Permission denied
mkdir: cannot create directory ‘/sys/fs/cgroup/blkio/user.slice/user’: Permission denied
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/blkio/user.slice/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/cgroup.procs: No such file or directory
mkdir: cannot create directory ‘/sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/user’: No such file or directory
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/cgroup.procs: No such file or directory
mkdir: cannot create directory ‘/sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user’: No such file or directory
../../src/tests/isolator_tests.cpp:1274: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
  Actual: 256
Expected: 0
-bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user/cgroup.procs: No such file or directory
../../src/tests/isolator_tests.cpp:1283: Failure
Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
  Actual: 256
Expected: 0
[  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess (763 ms)
[----------] 1 test from UserCgroupIsolatorTest/1 (763 ms total)
[----------] 1 test from UserCgroupIsolatorTest/2, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup
../../src/tests/isolator_tests.cpp:1200: Failure
isolator: Failed to create PerfEvent isolator, invalid events: { cpu-cycles }
[  FAILED  ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess (6 ms)
[----------] 1 test from UserCgroupIsolatorTest/2 (6 ms total)

Issue Links

• depends upon: MESOS-2834 Support different perf output formats
• is blocked by: MESOS-3123 DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged fails & crashes
• is cloned by: MESOS-3293 Failing ROOT_ tests on CentOS 7.1 - LimitedCpuIsolatorTest
• is duplicated by: MESOS-3047 Failing ROOT_DOCKER_Launch_Executor tests
• is duplicated by: MESOS-3048 Failing ROOT_DOCKER_NC_PortMapping
• is duplicated by: MESOS-3049 Failing CGROUPS_ROOT_PerfRollForward test
• relates to: MESOS-3079 `sudo make distcheck` fails on Ubuntu 14.04 (and possibly other OSes too)