Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-1195

systemd.slice + cgroup enablement fails in multiple ways.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.21.0
    • Component/s: containerization
    • Labels:
      None
    • Target Version/s:
    • Epic Link:
    • Sprint:
      Mesos Q3 Sprint 6
    • Story Points:
      3

      Description

      When attempting to configure mesos to use systemd slices on a 'rawhide/f21' machine, it fails creating the isolator:

      I0407 12:39:28.035354 14916 containerizer.cpp:180] Using isolation: cgroups/cpu,cgroups/mem
      Failed to create a containerizer: Could not create isolator cgroups/cpu: Failed to create isolator: The cpu subsystem is co-mounted at /sys/fs/cgroup/cpu with other subsytems

      ------ details ------
      /sys/fs/cgroup
      total 0
      drwxr-xr-x. 12 root root 280 Mar 18 08:47 .
      drwxr-xr-x. 6 root root 0 Mar 18 08:47 ..
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 blkio
      lrwxrwxrwx. 1 root root 11 Mar 18 08:47 cpu -> cpu,cpuacct
      lrwxrwxrwx. 1 root root 11 Mar 18 08:47 cpuacct -> cpu,cpuacct
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 cpu,cpuacct
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 cpuset
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 devices
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 freezer
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 hugetlb
      drwxr-xr-x. 3 root root 0 Apr 3 11:26 memory
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 net_cls
      drwxr-xr-x. 2 root root 0 Mar 18 08:47 perf_event
      drwxr-xr-x. 4 root root 0 Mar 18 08:47 systemd

        Activity

        Hide
        tstclair Timothy St. Clair added a comment -

        FWIW the logic fails in cgroups.cpp @:

        if (attached.get().size() > 1)

        { return Error("The " + subsystem + " subsystem is co-mounted at " + hierarchy + " with other subsytems"); }

        This logic has no comment, and doesn't really make sense to me, so perhaps a comment is in order why it exists at all

        Show
        tstclair Timothy St. Clair added a comment - FWIW the logic fails in cgroups.cpp @: if (attached.get().size() > 1) { return Error("The " + subsystem + " subsystem is co-mounted at " + hierarchy + " with other subsytems"); } This logic has no comment, and doesn't really make sense to me, so perhaps a comment is in order why it exists at all
        Hide
        nekto0n Nikita Vetoshkin added a comment -

        Are you trying to make mesos manipulate processes (put/remove) in cgroups, managed by systemd?

        Show
        nekto0n Nikita Vetoshkin added a comment - Are you trying to make mesos manipulate processes (put/remove) in cgroups, managed by systemd?
        Hide
        tstclair Timothy St. Clair added a comment -

        Correct, but the issue is due to the afore mentioned logic.

        Show
        tstclair Timothy St. Clair added a comment - Correct, but the issue is due to the afore mentioned logic.
        Hide
        nekto0n Nikita Vetoshkin added a comment -

        Is it a good idea? I mean doesn't systemd demand using dbus api to communicate with it?
        E.g. docker is using systemd's scopes to manage resources used by containers.

        Show
        nekto0n Nikita Vetoshkin added a comment - Is it a good idea? I mean doesn't systemd demand using dbus api to communicate with it? E.g. docker is using systemd's scopes to manage resources used by containers.
        Hide
        tstclair Timothy St. Clair added a comment -

        It is not required, "for some time", to leverage the dbus interface.
        There are tickets to address that separately.

        Show
        tstclair Timothy St. Clair added a comment - It is not required, "for some time", to leverage the dbus interface. There are tickets to address that separately.
        Hide
        nekto0n Nikita Vetoshkin added a comment -

        I see, thanks!

        Show
        nekto0n Nikita Vetoshkin added a comment - I see, thanks!
        Hide
        idownes Ian Downes added a comment -

        The reasoning behind not supporting multiple subsystems mounted together was so different Isolators could take ownership of hierarchies and not stomp on each other. However, cpu and cpuacct are so closely related that they are generally used together (hence systemd's layout) and indeed the CgroupsCpuIsolator uses both. It can readily be updated to support this configuration.

        Show
        idownes Ian Downes added a comment - The reasoning behind not supporting multiple subsystems mounted together was so different Isolators could take ownership of hierarchies and not stomp on each other. However, cpu and cpuacct are so closely related that they are generally used together (hence systemd's layout) and indeed the CgroupsCpuIsolator uses both. It can readily be updated to support this configuration.
        Hide
        tstclair Timothy St. Clair added a comment -

        I'll patch shortly to add the single exception.

        Show
        tstclair Timothy St. Clair added a comment - I'll patch shortly to add the single exception.
        Hide
        tstclair Timothy St. Clair added a comment - - edited

        There were a couple of issues.
        reviews.apache.org/r/21799/

        Show
        tstclair Timothy St. Clair added a comment - - edited There were a couple of issues. reviews.apache.org/r/21799/
        Hide
        tstclair Timothy St. Clair added a comment -

        Jie Yu ^ FYI.

        Show
        tstclair Timothy St. Clair added a comment - Jie Yu ^ FYI.
        Hide
        tstclair Timothy St. Clair added a comment - - edited

        DONE reviews.apache.org/r/22977/
        DONE reviews.apache.org/r/22979

        Show
        tstclair Timothy St. Clair added a comment - - edited DONE reviews.apache.org/r/22977/ DONE reviews.apache.org/r/22979
        Hide
        vinodkone Vinod Kone added a comment -

        moving this back to "in progress" because this is awaiting updates from tim.

        Show
        vinodkone Vinod Kone added a comment - moving this back to "in progress" because this is awaiting updates from tim.
        Hide
        tstclair Timothy St. Clair added a comment -

        I'm planning to circle back to this after creation of 0.20 rpms, for the next point release.

        Show
        tstclair Timothy St. Clair added a comment - I'm planning to circle back to this after creation of 0.20 rpms, for the next point release.
        Hide
        tstclair Timothy St. Clair added a comment -

        Back on *this.

        Show
        tstclair Timothy St. Clair added a comment - Back on *this.
        Hide
        tstclair Timothy St. Clair added a comment -

        reviews.apache.org/r/25695/

        Show
        tstclair Timothy St. Clair added a comment - reviews.apache.org/r/25695/
        Hide
        tstclair Timothy St. Clair added a comment -

        Bhuvan Arumugam please eval for 0.20.1 inclusion.

        Show
        tstclair Timothy St. Clair added a comment - Bhuvan Arumugam please eval for 0.20.1 inclusion.
        Hide
        bhuvan Bhuvan Arumugam added a comment -

        Timothy St. Clair the patch is still not reviewed. i'm going to offload it to 0.21.0.

        Show
        bhuvan Bhuvan Arumugam added a comment - Timothy St. Clair the patch is still not reviewed. i'm going to offload it to 0.21.0.
        Hide
        bhuvan Bhuvan Arumugam added a comment -

        moving it to 0.21.0, as discussed in reviewboard.
        http://reviews.apache.org/r/25695/

        Show
        bhuvan Bhuvan Arumugam added a comment - moving it to 0.21.0, as discussed in reviewboard. http://reviews.apache.org/r/25695/
        Hide
        jieyu Jie Yu added a comment -

        commit 31337348cef29719890bffb59fbf8df8b18b39d0
        Author: Jie Yu <yujie.jay@gmail.com>
        Date: Fri Sep 19 17:18:38 2014 -0700

        Allowed co-mounted cgroup subsystems to enable Mesos on machines with
        systemd.

        Review: https://reviews.apache.org/r/25858

        Show
        jieyu Jie Yu added a comment - commit 31337348cef29719890bffb59fbf8df8b18b39d0 Author: Jie Yu <yujie.jay@gmail.com> Date: Fri Sep 19 17:18:38 2014 -0700 Allowed co-mounted cgroup subsystems to enable Mesos on machines with systemd. Review: https://reviews.apache.org/r/25858

          People

          • Assignee:
            tstclair Timothy St. Clair
            Reporter:
            tstclair Timothy St. Clair
            Shepherd:
            Jie Yu
          • Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development

                Agile