[MESOS-4035] UserCgroupIsolatorTest.ROOT_CGROUPS_UserCgroup fails on CentOS 6.6

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed

Environment

CentOS6.6

• Target Version/s: 0.27.0
• Epic Link: flaky
• Sprint: Mesosphere Sprint 26
• Assignee: Jan Schlicht
• Reporter: Gilbert Song
• Shepherd: Till Toenshoff
• Created: 02/Dec/15 01:48
• Updated: 14/Jan/16 12:28
• Resolved: 14/Jan/16 12:28

Description

`ROOT_CGROUPS_UserCgroup` on CentOS6.6 with 0.26rc3. The environment setup on CentOS6.6 is based on latest update of /docs/getting-started.md. Either using devtoolset-2 or devtoolset-3 returns the same failure.

If running `sudo ./bin/mesos-tests.sh --gtest_filter="ROOT_CGROUPS_UserCgroup"`, it would return failures as following log:

[==========] Running 3 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 1 test from UserCgroupIsolatorTest/0, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup
../../src/tests/mesos.cpp:722: Failure
cgroups::mount(hierarchy, subsystem): '/tmp/mesos_test_cgroup/perf_event' already exists in the file system
-------------------------------------------------------------
We cannot run any cgroups tests that require
a hierarchy with subsystem 'perf_event'
because we failed to find an existing hierarchy
or create a new one (tried '/tmp/mesos_test_cgroup/perf_event').
You can either remove all existing
hierarchies, or disable this test case
(i.e., --gtest_filter=-UserCgroupIsolatorTest/0.*).
-------------------------------------------------------------
../../src/tests/mesos.cpp:776: Failure
cgroups: '/tmp/mesos_test_cgroup/perf_event' is not a valid hierarchy
[  FAILED  ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess (1 ms)
[----------] 1 test from UserCgroupIsolatorTest/0 (1 ms total)

[----------] 1 test from UserCgroupIsolatorTest/1, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup
../../src/tests/mesos.cpp:722: Failure
cgroups::mount(hierarchy, subsystem): '/tmp/mesos_test_cgroup/perf_event' already exists in the file system
-------------------------------------------------------------
We cannot run any cgroups tests that require
a hierarchy with subsystem 'perf_event'
because we failed to find an existing hierarchy
or create a new one (tried '/tmp/mesos_test_cgroup/perf_event').
You can either remove all existing
hierarchies, or disable this test case
(i.e., --gtest_filter=-UserCgroupIsolatorTest/1.*).
-------------------------------------------------------------
../../src/tests/mesos.cpp:776: Failure
cgroups: '/tmp/mesos_test_cgroup/perf_event' is not a valid hierarchy
[  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess (4 ms)
[----------] 1 test from UserCgroupIsolatorTest/1 (5 ms total)

[----------] 1 test from UserCgroupIsolatorTest/2, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup
../../src/tests/mesos.cpp:722: Failure
cgroups::mount(hierarchy, subsystem): '/tmp/mesos_test_cgroup/perf_event' already exists in the file system
-------------------------------------------------------------
We cannot run any cgroups tests that require
a hierarchy with subsystem 'perf_event'
because we failed to find an existing hierarchy
or create a new one (tried '/tmp/mesos_test_cgroup/perf_event').
You can either remove all existing
hierarchies, or disable this test case
(i.e., --gtest_filter=-UserCgroupIsolatorTest/2.*).
-------------------------------------------------------------
../../src/tests/mesos.cpp:776: Failure
cgroups: '/tmp/mesos_test_cgroup/perf_event' is not a valid hierarchy
[  FAILED  ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess (2 ms)
[----------] 1 test from UserCgroupIsolatorTest/2 (2 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 3 test cases ran. (349 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess
[  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
[  FAILED  ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess

 3 FAILED TESTS

If running it with `sudo ./bin/mesos-tests.sh --gtest_filter="ROOT_CGROUPS_UserCgroup" --gtest_repeat=-1 --gtest_break_on_failure`, it returned a segmentation fault (at iteration 1) as following log:

[==========] Running 3 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 1 test from UserCgroupIsolatorTest/0, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess
userdel: user 'mesos.test.unprivileged.user' does not exist
[ RUN      ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup
../../src/tests/mesos.cpp:722: Failure
cgroups::mount(hierarchy, subsystem): '/tmp/mesos_test_cgroup/perf_event' already exists in the file system
-------------------------------------------------------------
We cannot run any cgroups tests that require
a hierarchy with subsystem 'perf_event'
because we failed to find an existing hierarchy
or create a new one (tried '/tmp/mesos_test_cgroup/perf_event').
You can either remove all existing
hierarchies, or disable this test case
(i.e., --gtest_filter=-UserCgroupIsolatorTest/0.*).
-------------------------------------------------------------
*** Aborted at 1449018895 (unix time) try "date -d @1449018895" if you are using GNU date ***
PC: @          0x152039f testing::UnitTest::AddTestPartResult()
*** SIGSEGV (@0x0) received by PID 2930 (TID 0x7f9a07d92840) from PID 0; stack trace: ***
    @     0x7f9a0194d790 (unknown)
    @          0x152039f testing::UnitTest::AddTestPartResult()
    @          0x151ff0e testing::internal::AssertHelper::operator=()
    @           0xed245f mesos::internal::tests::ContainerizerTest<>::SetUp()
    @          0x155a9a3 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @          0x1547f51 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @          0x15281d3 testing::Test::Run()
    @          0x1528e9b testing::TestInfo::Run()
    @          0x15295e7 testing::TestCase::Run()
    @          0x1530d42 testing::internal::UnitTestImpl::RunAllTests()
    @          0x1558163 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @          0x1549fc1 testing::internal::HandleExceptionsInMethodIfSupported<>()
    @          0x15309fb testing::UnitTest::Run()
    @           0xc808d1 RUN_ALL_TESTS()
    @           0xc7f306 main
    @     0x7f9a007b5d5d __libc_start_main
    @           0x783e79 (unknown)

Both of these two failure cases occur 100%, not occasionally.

Issue Links

• is duplicated by: MESOS-4038 SlaveRecoveryTests, UserCgroupIsolatorTests fail on CentOS 6.6