Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3050

Failing ROOT_ tests on CentOS 7.1

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • 0.23.0, 0.24.0
    • None
    • containerization, docker, test
    • CentOS Linux release 7.1.1503
      0.24.0

    • Mesosphere Sprint 16
    • 5

    Description

      Running `sudo make check` on CentOS 7.1 for Mesos 0.23.0-rc3 causes several several failures/errors:

      [ RUN      ] DockerTest.ROOT_DOCKER_CheckPortResource
      ../../src/tests/docker_tests.cpp:303: Failure
      (run).failure(): Container exited on error: exited with status 1
      [  FAILED  ] DockerTest.ROOT_DOCKER_CheckPortResource (709 ms)
      

      ...

      [ RUN      ] PerfEventIsolatorTest.ROOT_CGROUPS_Sample
      ../../src/tests/isolator_tests.cpp:837: Failure
      isolator: Failed to create PerfEvent isolator, invalid events: { cycles, task-clock }
      [  FAILED  ] PerfEventIsolatorTest.ROOT_CGROUPS_Sample (9 ms)
      [----------] 1 test from PerfEventIsolatorTest (9 ms total)
         
      [----------] 2 tests from SharedFilesystemIsolatorTest
      [ RUN      ] SharedFilesystemIsolatorTest.ROOT_RelativeVolume
      + mount -n --bind /tmp/SharedFilesystemIsolatorTest_ROOT_RelativeVolume_4yTEAC/var/tmp /var/tmp
      + touch /var/tmp/492407e1-5dec-4b34-8f2f-130430f41aac
      ../../src/tests/isolator_tests.cpp:1001: Failure
      Value of: os::exists(file)
        Actual: true
      Expected: false
      [  FAILED  ] SharedFilesystemIsolatorTest.ROOT_RelativeVolume (92 ms)
      [ RUN      ] SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume
      + mount -n --bind /tmp/SharedFilesystemIsolatorTest_ROOT_AbsoluteVolume_OwYrXK /var/tmp
      + touch /var/tmp/7de712aa-52eb-4976-b0f9-32b6a006418d
      ../../src/tests/isolator_tests.cpp:1086: Failure
      Value of: os::exists(path::join(containerPath, filename))
        Actual: true
      Expected: false
      [  FAILED  ] SharedFilesystemIsolatorTest.ROOT_AbsoluteVolume (100 ms)
      

      ...

      [----------] 1 test from UserCgroupIsolatorTest/0, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess
      userdel: user 'mesos.test.unprivileged.user' does not exist
      [ RUN      ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup
      -bash: /sys/fs/cgroup/blkio/user.slice/cgroup.procs: Permission denied
      mkdir: cannot create directory ‘/sys/fs/cgroup/blkio/user.slice/user’: Permission denied
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/blkio/user.slice/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/memory/mesos/bbf8c8f0-3d67-40df-a269-b3dc6a9597aa/cgroup.procs: Permission denied
      -bash: /sys/fs/cgroup/cpuacct,cpu/user.slice/cgroup.procs: No such file or directory
      mkdir: cannot create directory ‘/sys/fs/cgroup/cpuacct,cpu/user.slice/user’: No such file or directory
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/cpuacct,cpu/user.slice/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/cgroup.procs: No such file or directory
      mkdir: cannot create directory ‘/sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user’: No such file or directory
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      [  FAILED  ] UserCgroupIsolatorTest/0.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsMemIsolatorProcess (1034 ms)
      [----------] 1 test from UserCgroupIsolatorTest/0 (1034 ms total)
      [----------] 1 test from UserCgroupIsolatorTest/1, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
      userdel: user 'mesos.test.unprivileged.user' does not exist
      [ RUN      ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup
      -bash: /sys/fs/cgroup/blkio/user.slice/cgroup.procs: Permission denied
      mkdir: cannot create directory ‘/sys/fs/cgroup/blkio/user.slice/user’: Permission denied
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/blkio/user.slice/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/cgroup.procs: No such file or directory
      mkdir: cannot create directory ‘/sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/user’: No such file or directory
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/cpuacct,cpu/mesos/eeeb99f0-7c5c-4185-869d-635d51dcc6e1/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/cgroup.procs: No such file or directory
      mkdir: cannot create directory ‘/sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user’: No such file or directory
      ../../src/tests/isolator_tests.cpp:1274: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'mkdir " + path::join(flags.cgroups_hierarchy, userCgroup) + "'")
        Actual: 256
      Expected: 0
      -bash: /sys/fs/cgroup/name=systemd/user.slice/user-2004.slice/session-3865.scope/user/cgroup.procs: No such file or directory
      ../../src/tests/isolator_tests.cpp:1283: Failure
      Value of: os::system( "su - " + UNPRIVILEGED_USERNAME + " -c 'echo $$ >" + path::join(flags.cgroups_hierarchy, userCgroup, "cgroup.procs") + "'")
        Actual: 256
      Expected: 0
      [  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess (763 ms)
      [----------] 1 test from UserCgroupIsolatorTest/1 (763 ms total)
      [----------] 1 test from UserCgroupIsolatorTest/2, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess
      userdel: user 'mesos.test.unprivileged.user' does not exist
      [ RUN      ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup
      ../../src/tests/isolator_tests.cpp:1200: Failure
      isolator: Failed to create PerfEvent isolator, invalid events: { cpu-cycles }
      [  FAILED  ] UserCgroupIsolatorTest/2.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsPerfEventIsolatorProcess (6 ms)
      [----------] 1 test from UserCgroupIsolatorTest/2 (6 ms total)
      

      Attachments

        1. 20150818-mesos-tests.log
          9.84 MB
          Marco Massenzio
        2. ROOT_tests.log
          9.83 MB
          Marco Massenzio

        Issue Links

          Activity

            adam-mesos Adam B added a comment -

            Also,

            [ RUN      ] CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled
            ../../src/tests/cgroups_tests.cpp:149: Failure
            cgroups::mount(hierarchy, subsystem): 'cpu' is already attached to another hierarchy
            -------------------------------------------------------------
            We cannot run any cgroups tests that require
            a hierarchy with subsystem 'cpu'
            because we failed to find an existing hierarchy
            or create a new one (tried '/sys/fs/cgroup/cpu,cpuacct/cpu').
            You can either remove all existing
            hierarchies, or disable this test case
            (i.e., --gtest_filter=-CgroupsAnyHierarchyTest.*).
            -------------------------------------------------------------
            F0715 01:00:39.523605  2778 cgroups_tests.cpp:183] CHECK_SOME(cgroups): '/sys/fs/cgroup/cpu,cpuacct/cpu' is not a valid hierarchy
            *** Check failure stack trace: ***
                @     0x7f097b47916e  google::LogMessage::Fail()
                @     0x7f097b4790cd  google::LogMessage::SendToLog()
                @     0x7f097b478ade  google::LogMessage::Flush()
                @     0x7f097b47b812  google::LogMessageFatal::~LogMessageFatal()
                @           0x881376  _CheckFatal::~_CheckFatal()
                @          0x11807ef  mesos::internal::tests::CgroupsAnyHierarchyTest::TearDown()
                @          0x120004f  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fb25a  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e36cc  testing::Test::Run()
                @          0x11e3da6  testing::TestInfo::Run()
                @          0x11e432e  testing::TestCase::Run()
                @          0x11e90aa  testing::internal::UnitTestImpl::RunAllTests()
                @          0x1200ec1  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fbf3d  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e7f5a  testing::UnitTest::Run()
                @           0xcfd614  main
                @     0x7f0976ac3af5  __libc_start_main
                @           0x87e109  (unknown)
            make[3]: *** [check-local] Aborted (core dumped)
            
            adam-mesos Adam B added a comment - Also, [ RUN ] CgroupsAnyHierarchyTest.ROOT_CGROUPS_Enabled ../../src/tests/cgroups_tests.cpp:149: Failure cgroups::mount(hierarchy, subsystem): 'cpu' is already attached to another hierarchy ------------------------------------------------------------- We cannot run any cgroups tests that require a hierarchy with subsystem 'cpu' because we failed to find an existing hierarchy or create a new one (tried '/sys/fs/cgroup/cpu,cpuacct/cpu' ). You can either remove all existing hierarchies, or disable this test case (i.e., --gtest_filter=-CgroupsAnyHierarchyTest.*). ------------------------------------------------------------- F0715 01:00:39.523605 2778 cgroups_tests.cpp:183] CHECK_SOME(cgroups): '/sys/fs/cgroup/cpu,cpuacct/cpu' is not a valid hierarchy *** Check failure stack trace: *** @ 0x7f097b47916e google::LogMessage::Fail() @ 0x7f097b4790cd google::LogMessage::SendToLog() @ 0x7f097b478ade google::LogMessage::Flush() @ 0x7f097b47b812 google::LogMessageFatal::~LogMessageFatal() @ 0x881376 _CheckFatal::~_CheckFatal() @ 0x11807ef mesos::internal::tests::CgroupsAnyHierarchyTest::TearDown() @ 0x120004f testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fb25a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e36cc testing::Test::Run() @ 0x11e3da6 testing::TestInfo::Run() @ 0x11e432e testing::TestCase::Run() @ 0x11e90aa testing::internal::UnitTestImpl::RunAllTests() @ 0x1200ec1 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fbf3d testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e7f5a testing::UnitTest::Run() @ 0xcfd614 main @ 0x7f0976ac3af5 __libc_start_main @ 0x87e109 (unknown) make[3]: *** [check-local] Aborted (core dumped)
            tnachen Timothy Chen added a comment -

            The Port resources test on the test machine since there is also another process taking the port 10000 that it uses.

            Now the cgroup tests are really interesting, and I wonder why we never got into these problems.

            CgroupsAnyHierarchyTest failed to run because it assumes the subsystem name (e.g: cpu) matches the path that it retrieves from /proc/mounts and so forth, and uses that to get the base hierarchy. However, it actually returns /sys/fs/cgroup/cpuacct,cpu instead so it didn't correctly the right base hierarchy name and fails.

            The UserCgroupIsolator test is also a very interesting case, where the detection of the cpu subsystem also failed because it got a cpuacct,cpu subsystem entry in the /proc/pid/cgroups, but it actually is named cpu,cpuacct in the mounted name and therefore couldn't complete as well.

            tnachen Timothy Chen added a comment - The Port resources test on the test machine since there is also another process taking the port 10000 that it uses. Now the cgroup tests are really interesting, and I wonder why we never got into these problems. CgroupsAnyHierarchyTest failed to run because it assumes the subsystem name (e.g: cpu) matches the path that it retrieves from /proc/mounts and so forth, and uses that to get the base hierarchy. However, it actually returns /sys/fs/cgroup/cpuacct,cpu instead so it didn't correctly the right base hierarchy name and fails. The UserCgroupIsolator test is also a very interesting case, where the detection of the cpu subsystem also failed because it got a cpuacct,cpu subsystem entry in the /proc/pid/cgroups, but it actually is named cpu,cpuacct in the mounted name and therefore couldn't complete as well.
            tnachen Timothy Chen added a comment -

            I don't understand what's going on with SharedFileSystemIsolator actually, it seems to do a simple fork with CLONE_NEWNS, in the child mounts in a directory and make changes in that mounted directory. However, the changes are somehow visible on the host.
            I was able to repro it with unshare, so need to investigate what possibly can cause the mount namespace to not work

            tnachen Timothy Chen added a comment - I don't understand what's going on with SharedFileSystemIsolator actually, it seems to do a simple fork with CLONE_NEWNS, in the child mounts in a directory and make changes in that mounted directory. However, the changes are somehow visible on the host. I was able to repro it with unshare, so need to investigate what possibly can cause the mount namespace to not work
            tnachen Timothy Chen added a comment -

            commit 7515cc9de2426ef80b50635a58e9c1442975fb59
            Author: Timothy Chen <tnachen@apache.org>
            Date: Wed Jul 15 11:29:31 2015 -0700

            Disable SharedFilesystemIsolator tests.

            Review: https://reviews.apache.org/r/36517

            tnachen Timothy Chen added a comment - commit 7515cc9de2426ef80b50635a58e9c1442975fb59 Author: Timothy Chen <tnachen@apache.org> Date: Wed Jul 15 11:29:31 2015 -0700 Disable SharedFilesystemIsolator tests. Review: https://reviews.apache.org/r/36517
            tnachen Timothy Chen added a comment -

            commit 21ed374dbfb5eedbba147c7ec33f29bc5ba02a34
            Author: Timothy Chen <tnachen@apache.org>
            Date: Wed Jul 15 12:59:46 2015 -0700

            Ignore warnings around removing unknown user in isolator tests.

            commit 67c994d1a432e590295d3471993c1dc3a59ebd70
            Author: Timothy Chen <tnachen@apache.org>
            Date: Wed Jul 15 12:56:53 2015 -0700

            Fix container cgroup detection in isolator tests.

            commit d975e45477b7fbddb298afeb96079f68d11a460e
            Author: Timothy Chen <tnachen@apache.org>
            Date: Wed Jul 15 12:56:25 2015 -0700

            Improve base hierarchy detection in cgroup tests.

            tnachen Timothy Chen added a comment - commit 21ed374dbfb5eedbba147c7ec33f29bc5ba02a34 Author: Timothy Chen <tnachen@apache.org> Date: Wed Jul 15 12:59:46 2015 -0700 Ignore warnings around removing unknown user in isolator tests. commit 67c994d1a432e590295d3471993c1dc3a59ebd70 Author: Timothy Chen <tnachen@apache.org> Date: Wed Jul 15 12:56:53 2015 -0700 Fix container cgroup detection in isolator tests. commit d975e45477b7fbddb298afeb96079f68d11a460e Author: Timothy Chen <tnachen@apache.org> Date: Wed Jul 15 12:56:25 2015 -0700 Improve base hierarchy detection in cgroup tests.
            adam-mesos Adam B added a comment -

            tnachen filed a CentOS error for the cgroups subsystem name issue: https://bugs.centos.org/view.php?id=9078

            I'm not convinced this is a blocker for 0.23.0, since previous releases presumably failed in the same way on CentOS 7.x
            We could disable the tests or find some hacky workaround, but is that reason enough to call for a new release candidate?

            adam-mesos Adam B added a comment - tnachen filed a CentOS error for the cgroups subsystem name issue: https://bugs.centos.org/view.php?id=9078 I'm not convinced this is a blocker for 0.23.0, since previous releases presumably failed in the same way on CentOS 7.x We could disable the tests or find some hacky workaround, but is that reason enough to call for a new release candidate?
            pbrett Paul Brett added a comment -

            PerfEventIsolatorTest is due to incompatible perf output version, will be fixed by MESOS-2834.

            pbrett Paul Brett added a comment - PerfEventIsolatorTest is due to incompatible perf output version, will be fixed by MESOS-2834 .
            adam-mesos Adam B added a comment -

            Merging in the errors from MESOS-3047..3049

            [ RUN      ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor
            ../../src/tests/docker_containerizer_tests.cpp:251: Failure
            Failed to wait 30secs for docker.get()->rm(container.id, true)
            [  FAILED  ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor (54563 ms)
            [ RUN      ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged
            ../../src/tests/docker_containerizer_tests.cpp:618: Failure
            Value of: statusRunning.get().state()
              Actual: TASK_LOST
            Expected: TASK_RUNNING
            ../../src/tests/docker_containerizer_tests.cpp:619: Failure
            Failed to wait 1mins for statusFinished
            ../../src/tests/docker_containerizer_tests.cpp:610: Failure
            Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))...
                     Expected: to be called twice
                       Actual: called once - unsatisfied and active
            F0714 23:14:43.266775  9319 logging.cpp:57] RAW: Pure virtual method called
                @     0x7ffb4f2d016e  google::LogMessage::Fail()
                @     0x7ffb4f2d5650  google::RawLog__()
                @     0x7ffb4e8294c6  __cxa_pure_virtual
                @           0x888288  mesos::internal::tests::Cluster::Slaves::shutdown()
                @           0x888050  mesos::internal::tests::Cluster::Slaves::~Slaves()
                @           0x889af0  mesos::internal::tests::Cluster::~Cluster()
                @           0x889b61  mesos::internal::tests::MesosTest::~MesosTest()
                @           0xa4e5b1  mesos::internal::tests::DockerContainerizerTest::~DockerContainerizerTest()
                @           0xa7cd65  mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test::~DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test()
                @           0xa7cd94  mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test::~DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test()
                @          0x11f722e  testing::Test::DeleteSelf_()
                @          0x120004f  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fb25a  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e3df3  testing::TestInfo::Run()
                @          0x11e432e  testing::TestCase::Run()
                @          0x11e90aa  testing::internal::UnitTestImpl::RunAllTests()
                @          0x1200ec1  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fbf3d  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e7f5a  testing::UnitTest::Run()
                @           0xcfd614  main
                @     0x7ffb4a91aaf5  __libc_start_main
                @           0x87e109  (unknown)
            make[3]: *** [check-local] Aborted (core dumped)
            

            ...

            [ RUN      ] MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward
            I0715 00:28:42.600359 17919 exec.cpp:132] Version: 0.23.0
            I0715 00:28:42.609073 17983 exec.cpp:206] Executor registered on slave 20150715-002842-201330860-41108-794-S0
            Registered executor on srv2.hw.ca1.mesosphere.com
            Starting task b675699c-8427-4934-8bb5-62f91f3d74d0
            Forked command at 17988
            sh -c 'sleep 1000'
            ../../src/tests/slave_recovery_tests.cpp:3446: Failure
            containerizer2: Could not create isolator cgroups/perf_event: Failed to create PerfEvent isolator, invalid events: { cycles, task-clock }
            ../../3rdparty/libprocess/include/process/gmock.hpp:353: Failure
            Actual function call count doesn't match EXPECT_CALL(filter->mock, filter(testing::A<const DispatchEvent&>()))...
                Expected args: dispatch matcher (1-byte object <04>, 16-byte object <29-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>)
                     Expected: to be called once
                       Actual: never called - unsatisfied and active
            [  FAILED  ] MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward (462 ms)
            

            ...

            [ RUN      ] DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping
            ../../src/tests/docker_containerizer_tests.cpp:2226: Failure
            Value of: statusRunning.get().state()
              Actual: TASK_FAILED
            Expected: TASK_RUNNING
            ../../src/tests/docker_containerizer_tests.cpp:2229: Failure
            Value of: exists(docker, slaveId, containerId.get(), ContainerState::RUNNING)
              Actual: false
            Expected: true
            ../../src/tests/docker_containerizer_tests.cpp:2216: Failure
            Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))...
                     Expected: to be called at least twice
                       Actual: called once - unsatisfied and active
            F0715 00:01:27.612823 26144 logging.cpp:57] RAW: Pure virtual method called
                @     0x7fd8222f416e  google::LogMessage::Fail()
                @     0x7fd8222f9650  google::RawLog__()
                @     0x7fd82184d4c6  __cxa_pure_virtual
                @           0x888288  mesos::internal::tests::Cluster::Slaves::shutdown()
                @           0x888050  mesos::internal::tests::Cluster::Slaves::~Slaves()
                @           0x889af0  mesos::internal::tests::Cluster::~Cluster()
                @           0x889b61  mesos::internal::tests::MesosTest::~MesosTest()
                @           0xa4e5b1  mesos::internal::tests::DockerContainerizerTest::~DockerContainerizerTest()
                @           0xa7c3a5  mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test::~DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test()
                @           0xa7c3d4  mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test::~DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test()
                @          0x11f722e  testing::Test::DeleteSelf_()
                @          0x120004f  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fb25a  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e3df3  testing::TestInfo::Run()
                @          0x11e432e  testing::TestCase::Run()
                @          0x11e90aa  testing::internal::UnitTestImpl::RunAllTests()
                @          0x1200ec1  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
                @          0x11fbf3d  testing::internal::HandleExceptionsInMethodIfSupported<>()
                @          0x11e7f5a  testing::UnitTest::Run()
                @           0xcfd614  main
                @     0x7fd81d93eaf5  __libc_start_main
                @           0x87e109  (unknown)
            make[3]: *** [check-local] Aborted (core dumped)
            
            adam-mesos Adam B added a comment - Merging in the errors from MESOS-3047 ..3049 [ RUN ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor ../../src/tests/docker_containerizer_tests.cpp:251: Failure Failed to wait 30secs for docker.get()->rm(container.id, true ) [ FAILED ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor (54563 ms) [ RUN ] DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged ../../src/tests/docker_containerizer_tests.cpp:618: Failure Value of: statusRunning.get().state() Actual: TASK_LOST Expected: TASK_RUNNING ../../src/tests/docker_containerizer_tests.cpp:619: Failure Failed to wait 1mins for statusFinished ../../src/tests/docker_containerizer_tests.cpp:610: Failure Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))... Expected: to be called twice Actual: called once - unsatisfied and active F0714 23:14:43.266775 9319 logging.cpp:57] RAW: Pure virtual method called @ 0x7ffb4f2d016e google::LogMessage::Fail() @ 0x7ffb4f2d5650 google::RawLog__() @ 0x7ffb4e8294c6 __cxa_pure_virtual @ 0x888288 mesos::internal::tests::Cluster::Slaves::shutdown() @ 0x888050 mesos::internal::tests::Cluster::Slaves::~Slaves() @ 0x889af0 mesos::internal::tests::Cluster::~Cluster() @ 0x889b61 mesos::internal::tests::MesosTest::~MesosTest() @ 0xa4e5b1 mesos::internal::tests::DockerContainerizerTest::~DockerContainerizerTest() @ 0xa7cd65 mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test::~DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test() @ 0xa7cd94 mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test::~DockerContainerizerTest_ROOT_DOCKER_Launch_Executor_Bridged_Test() @ 0x11f722e testing::Test::DeleteSelf_() @ 0x120004f testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fb25a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e3df3 testing::TestInfo::Run() @ 0x11e432e testing::TestCase::Run() @ 0x11e90aa testing::internal::UnitTestImpl::RunAllTests() @ 0x1200ec1 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fbf3d testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e7f5a testing::UnitTest::Run() @ 0xcfd614 main @ 0x7ffb4a91aaf5 __libc_start_main @ 0x87e109 (unknown) make[3]: *** [check-local] Aborted (core dumped) ... [ RUN ] MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward I0715 00:28:42.600359 17919 exec.cpp:132] Version: 0.23.0 I0715 00:28:42.609073 17983 exec.cpp:206] Executor registered on slave 20150715-002842-201330860-41108-794-S0 Registered executor on srv2.hw.ca1.mesosphere.com Starting task b675699c-8427-4934-8bb5-62f91f3d74d0 Forked command at 17988 sh -c 'sleep 1000' ../../src/tests/slave_recovery_tests.cpp:3446: Failure containerizer2: Could not create isolator cgroups/perf_event: Failed to create PerfEvent isolator, invalid events: { cycles, task-clock } ../../3rdparty/libprocess/include/process/gmock.hpp:353: Failure Actual function call count doesn't match EXPECT_CALL(filter->mock, filter(testing::A< const DispatchEvent&>()))... Expected args: dispatch matcher (1- byte object <04>, 16- byte object <29-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00>) Expected: to be called once Actual: never called - unsatisfied and active [ FAILED ] MesosContainerizerSlaveRecoveryTest.CGROUPS_ROOT_PerfRollForward (462 ms) ... [ RUN ] DockerContainerizerTest.ROOT_DOCKER_NC_PortMapping ../../src/tests/docker_containerizer_tests.cpp:2226: Failure Value of: statusRunning.get().state() Actual: TASK_FAILED Expected: TASK_RUNNING ../../src/tests/docker_containerizer_tests.cpp:2229: Failure Value of: exists(docker, slaveId, containerId.get(), ContainerState::RUNNING) Actual: false Expected: true ../../src/tests/docker_containerizer_tests.cpp:2216: Failure Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(&driver, _))... Expected: to be called at least twice Actual: called once - unsatisfied and active F0715 00:01:27.612823 26144 logging.cpp:57] RAW: Pure virtual method called @ 0x7fd8222f416e google::LogMessage::Fail() @ 0x7fd8222f9650 google::RawLog__() @ 0x7fd82184d4c6 __cxa_pure_virtual @ 0x888288 mesos::internal::tests::Cluster::Slaves::shutdown() @ 0x888050 mesos::internal::tests::Cluster::Slaves::~Slaves() @ 0x889af0 mesos::internal::tests::Cluster::~Cluster() @ 0x889b61 mesos::internal::tests::MesosTest::~MesosTest() @ 0xa4e5b1 mesos::internal::tests::DockerContainerizerTest::~DockerContainerizerTest() @ 0xa7c3a5 mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test::~DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test() @ 0xa7c3d4 mesos::internal::tests::DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test::~DockerContainerizerTest_ROOT_DOCKER_NC_PortMapping_Test() @ 0x11f722e testing::Test::DeleteSelf_() @ 0x120004f testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fb25a testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e3df3 testing::TestInfo::Run() @ 0x11e432e testing::TestCase::Run() @ 0x11e90aa testing::internal::UnitTestImpl::RunAllTests() @ 0x1200ec1 testing::internal::HandleSehExceptionsInMethodIfSupported<>() @ 0x11fbf3d testing::internal::HandleExceptionsInMethodIfSupported<>() @ 0x11e7f5a testing::UnitTest::Run() @ 0xcfd614 main @ 0x7fd81d93eaf5 __libc_start_main @ 0x87e109 (unknown) make[3]: *** [check-local] Aborted (core dumped)
            adam-mesos Adam B added a comment -

            Pulled the already committed patches into 0.23.0-rc4. Deferring this issue to 0.24 for the rest of the failing tests.

            adam-mesos Adam B added a comment - Pulled the already committed patches into 0.23.0-rc4. Deferring this issue to 0.24 for the rest of the failing tests.

            ROOT Tests continue to fail in CentOS 7.x:

            [==========] 751 tests from 114 test cases ran. (215470 ms total)
            [  PASSED  ] 743 tests.
            [  FAILED  ] 8 tests, listed below:
            [  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
            [  FAILED  ] CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreaseRSS
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem
            [  FAILED  ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs
            
             8 FAILED TESTS
            

            Executed in a VirtualBox VM running on a Ubuntu 14.04 host:

            [centos-mesos-dev ~/mesos/build]$ cat /etc/os-release 
            NAME="CentOS Linux"
            VERSION="7 (Core)"
            ID="centos"
            ID_LIKE="rhel fedora"
            VERSION_ID="7"
            PRETTY_NAME="CentOS Linux 7 (Core)"
            ANSI_COLOR="0;31"
            CPE_NAME="cpe:/o:centos:centos:7"
            HOME_URL="https://www.centos.org/"
            BUG_REPORT_URL="https://bugs.centos.org/"
            
            CENTOS_MANTISBT_PROJECT="CentOS-7"
            CENTOS_MANTISBT_PROJECT_VERSION="7"
            REDHAT_SUPPORT_PRODUCT="centos"
            REDHAT_SUPPORT_PRODUCT_VERSION="7"
            
            marco-mesos Marco Massenzio added a comment - ROOT Tests continue to fail in CentOS 7.x: [==========] 751 tests from 114 test cases ran. (215470 ms total) [ PASSED ] 743 tests. [ FAILED ] 8 tests, listed below: [ FAILED ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess [ FAILED ] CgroupsAnyHierarchyMemoryPressureTest.ROOT_IncreaseRSS [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem [ FAILED ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs 8 FAILED TESTS Executed in a VirtualBox VM running on a Ubuntu 14.04 host: [centos-mesos-dev ~/mesos/build]$ cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

            I am also seeing the following failure (but NOT the others) in the Ubuntu 14.04 box (physical box):

            [ RUN      ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids
            ../../src/tests/containerizer/isolator_tests.cpp:731: Failure
            Value of: usage.get().threads()
              Actual: 2
            Expected: 1U
            Which is: 1
            [  FAILED  ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids (96 ms)
            [----------] 3 tests from LimitedCpuIsolatorTest (814 ms total)
            

            All tests run with sudo ./bin/mesos-tests.sh.

            marco-mesos Marco Massenzio added a comment - I am also seeing the following failure (but NOT the others) in the Ubuntu 14.04 box (physical box): [ RUN ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids ../../src/tests/containerizer/isolator_tests.cpp:731: Failure Value of: usage.get().threads() Actual: 2 Expected: 1U Which is: 1 [ FAILED ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids (96 ms) [----------] 3 tests from LimitedCpuIsolatorTest (814 ms total) All tests run with sudo ./bin/mesos-tests.sh .

            Attached verbose log of failed tests on CentOS 7

            marco-mesos Marco Massenzio added a comment - Attached verbose log of failed tests on CentOS 7
            jieyu Jie Yu added a comment -

            Looking at the logs of those filesystem isolator tests, the 'exec' fails after pivot_root. Since we're exec-ing a '/bin/sh' binary, one explanation might be the binary (or some dependency of it) are not in the test root filesystem.

            jieyu Jie Yu added a comment - Looking at the logs of those filesystem isolator tests, the 'exec' fails after pivot_root. Since we're exec-ing a '/bin/sh' binary, one explanation might be the binary (or some dependency of it) are not in the test root filesystem.
            jieyu Jie Yu added a comment -

            OK, I think I know the problem. In centos7.1, 'sh' is under '/usr/bin/sh', while on centos6 (the system I've been using), 'sh' is under '/bin/sh'.

            jieyu Jie Yu added a comment - OK, I think I know the problem. In centos7.1, 'sh' is under '/usr/bin/sh', while on centos6 (the system I've been using), 'sh' is under '/bin/sh'.

            hrumpf... great investigation jieyu!
            Is there an easy fix or does this require to 'introspect' the system at runtime?

            marco-mesos Marco Massenzio added a comment - hrumpf... great investigation jieyu ! Is there an easy fix or does this require to 'introspect' the system at runtime?
            jieyu Jie Yu added a comment -

            Pushed a fix. marco-mesos, let me know if the tests are still failing.

            commit 3ae937fb1c41bf858d7e37e5679da646fe93734b
            Author: Jie Yu <yujie.jay@gmail.com>
            Date: Mon Aug 17 12:53:08 2015 -0700

            Included /usr/bin/sh in the test root filesystem.

            Review: https://reviews.apache.org/r/37555

            commit bd4332c68aea3aaf8eac3ef3a15b72541084e0c4
            Author: Jie Yu <yujie.jay@gmail.com>
            Date: Mon Aug 17 12:47:52 2015 -0700

            Used execlp instead of execl to exec processes in Mesos.

            Review: https://reviews.apache.org/r/37547

            commit d7d3b52122613f536bcffe41a5f26132e99728af
            Author: Jie Yu <yujie.jay@gmail.com>
            Date: Mon Aug 17 12:47:41 2015 -0700

            Used execlp instead of execl to exec processes in libprocess.

            Review: https://reviews.apache.org/r/37546

            commit e70493a8acd3c6848bb9dbe7f7a72e694fe6cf07
            Author: Jie Yu <yujie.jay@gmail.com>
            Date: Mon Aug 17 12:47:31 2015 -0700

            Used execlp instead of execl to exec processes in stout.

            Review: https://reviews.apache.org/r/37545

            jieyu Jie Yu added a comment - Pushed a fix. marco-mesos , let me know if the tests are still failing. commit 3ae937fb1c41bf858d7e37e5679da646fe93734b Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:53:08 2015 -0700 Included /usr/bin/sh in the test root filesystem. Review: https://reviews.apache.org/r/37555 commit bd4332c68aea3aaf8eac3ef3a15b72541084e0c4 Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:52 2015 -0700 Used execlp instead of execl to exec processes in Mesos. Review: https://reviews.apache.org/r/37547 commit d7d3b52122613f536bcffe41a5f26132e99728af Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:41 2015 -0700 Used execlp instead of execl to exec processes in libprocess. Review: https://reviews.apache.org/r/37546 commit e70493a8acd3c6848bb9dbe7f7a72e694fe6cf07 Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:31 2015 -0700 Used execlp instead of execl to exec processes in stout. Review: https://reviews.apache.org/r/37545

            I've pulled the most recent master:

            $ git log -5 --abbrev-commit --author Jie
            
            commit 3ae937f
            Author: Jie Yu <yujie.jay@gmail.com>
            Date:   Mon Aug 17 12:53:08 2015 -0700
            
                Included /usr/bin/sh in the test root filesystem.
              
                Review: https://reviews.apache.org/r/37555
            
            commit bd4332c
            Author: Jie Yu <yujie.jay@gmail.com>
            Date:   Mon Aug 17 12:47:52 2015 -0700
            
                Used execlp instead of execl to exec processes in Mesos.
                
                Review: https://reviews.apache.org/r/37547
            
            commit d7d3b52
            Author: Jie Yu <yujie.jay@gmail.com>
            Date:   Mon Aug 17 12:47:41 2015 -0700
            
                Used execlp instead of execl to exec processes in libprocess.
                
                Review: https://reviews.apache.org/r/37546
            
            commit e70493a
            Author: Jie Yu <yujie.jay@gmail.com>
            Date:   Mon Aug 17 12:47:31 2015 -0700
            
                Used execlp instead of execl to exec processes in stout.
                
                Review: https://reviews.apache.org/r/37545
            
            commit ccc4c60
            Author: Jie Yu <yujie.jay@gmail.com>
            Date:   Mon Aug 17 12:46:29 2015 -0700
            
                Fixed a bug in port mapping tests due to os::shell refactor.
                
                Review: https://reviews.apache.org/r/37544
            

            and re-run make && make check followed by sudo ./bin/mesos-tests.sh --verbose - the tests are all still failing:

            [==========] 751 tests from 114 test cases ran. (211548 ms total)
            [  PASSED  ] 743 tests.
            [  FAILED  ] 8 tests, listed below:
            [  FAILED  ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids
            [  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem
            [  FAILED  ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs
            
             8 FAILED TESTS
              YOU HAVE 10 DISABLED TESTS
            

            I'll now get a bit more aggressive, clean up the entire build and re-run it again.
            BTW - this is CentOS 7, is it the same you tested your changes on?

            marco-mesos Marco Massenzio added a comment - I've pulled the most recent master: $ git log -5 --abbrev-commit --author Jie commit 3ae937f Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:53:08 2015 -0700 Included /usr/bin/sh in the test root filesystem. Review: https://reviews.apache.org/r/37555 commit bd4332c Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:52 2015 -0700 Used execlp instead of execl to exec processes in Mesos. Review: https://reviews.apache.org/r/37547 commit d7d3b52 Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:41 2015 -0700 Used execlp instead of execl to exec processes in libprocess. Review: https://reviews.apache.org/r/37546 commit e70493a Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:47:31 2015 -0700 Used execlp instead of execl to exec processes in stout. Review: https://reviews.apache.org/r/37545 commit ccc4c60 Author: Jie Yu <yujie.jay@gmail.com> Date: Mon Aug 17 12:46:29 2015 -0700 Fixed a bug in port mapping tests due to os::shell refactor. Review: https://reviews.apache.org/r/37544 and re-run make && make check followed by sudo ./bin/mesos-tests.sh --verbose - the tests are all still failing: [==========] 751 tests from 114 test cases ran. (211548 ms total) [ PASSED ] 743 tests. [ FAILED ] 8 tests, listed below: [ FAILED ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids [ FAILED ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem [ FAILED ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs 8 FAILED TESTS YOU HAVE 10 DISABLED TESTS I'll now get a bit more aggressive, clean up the entire build and re-run it again. BTW - this is CentOS 7, is it the same you tested your changes on?
            vinodkone Vinod Kone added a comment -

            Marco. I'm planning to cut an RC now. Is this a blocker?

            vinodkone Vinod Kone added a comment - Marco. I'm planning to cut an RC now. Is this a blocker?
            vinodkone Vinod Kone added a comment -

            I'm unsetting the target version for now.

            vinodkone Vinod Kone added a comment - I'm unsetting the target version for now.

            Hi,

            I'm at SFO away from computer. 

            I don't really know, I think we had the same issue in 0.23 too. 

            @artem @adamb what do you guys think?


            Sent from Mailbox

            On Tue, Aug 18, 2015 at 12:30 PM, Vinod Kone (JIRA) <jira@apache.org>

            marco-mesos Marco Massenzio added a comment - Hi, I'm at SFO away from computer.  I don't really know, I think we had the same issue in 0.23 too.  @artem @adamb what do you guys think? — Sent from Mailbox On Tue, Aug 18, 2015 at 12:30 PM, Vinod Kone (JIRA) <jira@apache.org>

            I can tell for CgroupsAnyHierarchyMemoryPressureTest it's not a blocker because the problem is in a test itself. I don't know why is it failing again, I'll take a look, but it shouldn't be a blocker.

            hartem Artem Harutyunyan added a comment - I can tell for CgroupsAnyHierarchyMemoryPressureTest it's not a blocker because the problem is in a test itself. I don't know why is it failing again, I'll take a look, but it shouldn't be a blocker.
            adam-mesos Adam B added a comment -

            vinodkone We released 0.23 with some failing ROOT tests, so if it wasn't a blocker for 0.23, it shouldn't need to block 0.24.
            That said, this is troubling, and we should fix it ASAP, even cherry-picking fixes into subsequent 0.24-rc's, if there are any.

            I wonder if we should disable these tests for the 0.24 release, so that others won't run into them and get confused.
            Then we can resolve this issue and work on each of the disabled tests as separate tickets.

            adam-mesos Adam B added a comment - vinodkone We released 0.23 with some failing ROOT tests, so if it wasn't a blocker for 0.23, it shouldn't need to block 0.24. That said, this is troubling, and we should fix it ASAP, even cherry-picking fixes into subsequent 0.24-rc's, if there are any. I wonder if we should disable these tests for the 0.24 release, so that others won't run into them and get confused. Then we can resolve this issue and work on each of the disabled tests as separate tickets.

            (full verbose logs attached)

            Completely cleaned the build, removed directory, clean pull from master (SHA: fb93d93) - same results, 9 failed tests:

            [==========] 751 tests from 114 test cases ran. (231218 ms total)
            [  PASSED  ] 742 tests.
            [  FAILED  ] 9 tests, listed below:
            [  FAILED  ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids
            [  FAILED  ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess
            [  FAILED  ] ContainerizerTest.ROOT_CGROUPS_BalloonFramework
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint
            [  FAILED  ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem
            [  FAILED  ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs
            
             9 FAILED TESTS
              YOU HAVE 10 DISABLED TESTS
            

            Platform: CentOS 7, kernel 3.10.0; Virtualbox VM.

            marco-mesos Marco Massenzio added a comment - (full verbose logs attached) Completely cleaned the build, removed directory, clean pull from master (SHA: fb93d93 ) - same results, 9 failed tests: [==========] 751 tests from 114 test cases ran. (231218 ms total) [ PASSED ] 742 tests. [ FAILED ] 9 tests, listed below: [ FAILED ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Pids_and_Tids [ FAILED ] UserCgroupIsolatorTest/1.ROOT_CGROUPS_UserCgroup, where TypeParam = mesos::internal::slave::CgroupsCpushareIsolatorProcess [ FAILED ] ContainerizerTest.ROOT_CGROUPS_BalloonFramework [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_ChangeRootFilesystem [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromSandbox [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHost [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_VolumeFromHostSandboxMountPoint [ FAILED ] LinuxFilesystemIsolatorTest.ROOT_PersistentVolumeWithRootFilesystem [ FAILED ] MesosContainerizerLaunchTest.ROOT_ChangeRootfs 9 FAILED TESTS YOU HAVE 10 DISABLED TESTS Platform: CentOS 7, kernel 3.10.0; Virtualbox VM.
            marco-mesos Marco Massenzio added a comment - - edited

            +1 to creating the RC with the tests disabled

            AND having Jiras created for each one of them - we can then figure out whether they should be:

            • fixed;
            • removed; OR
            • redesigned.

            In any event, these should be considered "blockers" for 1.0 - we've already kicked the can since 0.23, this really needs sorting out.

            Created the following Issues:
            MESOS-3292
            MESOS-3293
            MESOS-3294
            MESOS-3295
            MESOS-3296
            MESOS-3297

            marco-mesos Marco Massenzio added a comment - - edited +1 to creating the RC with the tests disabled AND having Jiras created for each one of them - we can then figure out whether they should be: fixed; removed; OR redesigned. In any event, these should be considered "blockers" for 1.0 - we've already kicked the can since 0.23, this really needs sorting out. Created the following Issues: MESOS-3292 MESOS-3293 MESOS-3294 MESOS-3295 MESOS-3296 MESOS-3297

            The failing errors reported here for 0.24-rc1 have now been reported individually in their own Jiras - we can close this one and continue progress in those, so that reporting fixes etc. is less confusing.

            marco-mesos Marco Massenzio added a comment - The failing errors reported here for 0.24-rc1 have now been reported individually in their own Jiras - we can close this one and continue progress in those, so that reporting fixes etc. is less confusing.

            People

              marco-mesos Marco Massenzio
              adam-mesos Adam B
              Vinod Kone Vinod Kone
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: