[MESOS-3978] Missing support for cpu.cfs_quota_us breaks test suite.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Labels: mesosphere

Environment

Debian 8, kernel 3.16.0-4-amd64, gcc 4.9.2

• Target Version/s: 0.26.0
• Sprint: Mesosphere Sprint 23
• Assignee: Till Toenshoff
• Reporter: Till Toenshoff
• Created: 22/Nov/15 17:41
• Updated: 26/Nov/18 12:35
• Resolved: 26/Nov/18 12:35

Description

[ RUN      ] LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs
F1122 17:26:40.585412 23162 isolator_tests.cpp:455] CHECK_SOME(isolator): Failed to find 'cpu.cfs_quota_us'. Your kernel might be too old to use the CFS cgroups feature.
*** Check failure stack trace: ***
    @     0x7f81e3fc42ba  google::LogMessage::Fail()
    @     0x7f81e3fc4216  google::LogMessage::SendToLog()
    @     0x7f81e3fc3c0c  google::LogMessage::Flush()
    @     0x7f81e3fc69c6  google::LogMessageFatal::~LogMessageFatal()
    @           0x962139  _CheckFatal::~_CheckFatal()
    @          0x1305e48  mesos::internal::tests::LimitedCpuIsolatorTest_ROOT_CGROUPS_Cfs_Test::TestBody()
    @          0x1439638  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @          0x1434544  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @          0x1415883  testing::Test::Run()
    @          0x1416011  testing::TestInfo::Run()
    @          0x141664c  testing::TestCase::Run()
    @          0x141cf9b  testing::internal::UnitTestImpl::RunAllTests()
    @          0x143a2c7  testing::internal::HandleSehExceptionsInMethodIfSupported<>()
    @          0x14350d0  testing::internal::HandleExceptionsInMethodIfSupported<>()
    @          0x141bccb  testing::UnitTest::Run()
    @           0xd324ac  RUN_ALL_TESTS()
    @           0xd3208a  main
    @     0x7f81dde20b45  (unknown)
    @           0x92ae59  (unknown)

This hints that our kernel does not support cpu.cfs_quota_us. For checking, I simply did a:

$ ls -l /sys/fs/cgroup/cpu/cpu.*
-rw-r--r-- 1 root root 0 Nov 22 17:28 /sys/fs/cgroup/cpu/cpu.shares

So obviously this kernel won't support this feature as it is not listed. A rather simple test in our test-suite could be used to dis/enable those tests.

$ ls -l /sys/fs/cgroup/cpu/cpu.cfs_quota_us
ls: cannot access /sys/fs/cgroup/cpu/cpu.cfs_quota_us: No such file or directory
$ echo $?
2

We should make sure that either the test itself does not break or that it is excluded in the test-runs when the system does not support this feature.

Vagrant generator script used:

export VAGRANT_CPUS=8
export VAGRANT_MEM=16384

cat << EOF > Vagrantfile
# -*- mode: ruby -*-" >
# vi: set ft=ruby :
Vagrant.configure(2) do |config|

  # Disable shared folder to prevent certain kernel module dependencies.
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.hostname = "debian82"

  config.vm.box = "bento/debian-8.2"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = ENV['VAGRANT_MEM']
    vb.cpus = ENV['VAGRANT_CPUS']
  end

  config.vm.provider "vmware_fusion" do |vb|
    vb.memory = ENV['VAGRANT_MEM']
    vb.cpus = ENV['VAGRANT_CPUS']
  end

  config.vm.provision "shell", inline: <<-SHELL

    # Update the entire system.
    sudo apt-get update

    # Install Mesos dependecies.
    sudo apt-get install -y openjdk-7-jdk autoconf libtool
    sudo apt-get install -y build-essential python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev

    # Install latest Docker.
    sudo wget -qO- https://get.docker.com/ | sh

    # Enable memory and swap cgroups.
    sudo echo "GRUB_CMDLINE_LINUX_DEFAULT=\"cgroup_enable=memory swapaccount=1\"" >>/etc/default/grub
    sudo grub-mkconfig -o /boot/grub/grub.cfg "$@"
  SHELL
end
EOF

vagrant up
vagrant reload

vagrant ssh -c "
sudo docker info

git clone  https://github.com/apache/mesos.git mesos
cd mesos
git checkout -b 0.26.0-rc1 0.26.0-rc1

./bootstrap
mkdir build
cd build

../configure --disable-java --disable-python
GTEST_FILTER="" make check -j$VAGRANT_CPUS
sudo ./bin/mesos-tests.sh
"

Issue Links

• duplicates: MESOS-3964 LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs and LimitedCpuIsolatorTest.ROOT_CGROUPS_Cfs_Big_Quota fail on Debian 8.