Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2406

init scripts do not work reliable on systemd systems

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.2.0
    • Component/s: debian, Init scripts
    • Labels:
      None

      Description

      systemd is emulating the system v init system only up to some point.

      From the manual https://www.freedesktop.org/wiki/Software/systemd/Incompatibilities/

      systemd only stops running services. On traditional SysV a K link installed for shutdown was executed when going down regardless whether the service was started before or not. systemd is more strict here and does not stop service that weren't started in the first place.
      If systemd doesn't know which PID is the main PID of a service, it will not be able to track its runtime, and hence a service exiting on its own will not make systemd consider it stopped. Use the Red Hat "pidfile: " syntax in the SysV script header comment block to let systemd know which PID file (and hence PID) belongs to your service. Note that systemd cannot know if a SysV service is one of the kind where the runtime is defined by a specific process or whether it is one where there is none, hence the requirement of explicit configuration of a PID file in order to make systemd track the process lifetime.
      

      The main point is that if for instance the hadoop-yarn-resourcemanager dies, a puppet run will not restart the daemon, since the systemd still considers it running. This happens sometimes on debian8.

      I like to propose to support systemd unit files in addition to the existing sysv init scripts.

      And please do not bash systemd. It is a tremendous improvement of the Linux Boot process. We have to adapt.

        Activity

        Hide
        cos Konstantin Boudnik added a comment -

        That'd be pretty good to have. Especially, if we rework our recipes a little bit so that we can regenerate configurations and restart daemons separately from, say, formatting HDFS.

        Show
        cos Konstantin Boudnik added a comment - That'd be pretty good to have. Especially, if we rework our recipes a little bit so that we can regenerate configurations and restart daemons separately from, say, formatting HDFS.
        Hide
        oflebbe Olaf Flebbe added a comment -

        The system I presented in Budapest is doing this. http://github.com/oflebbe/inst

        But the system is outdated now, since Michael found a better way to use puppet for bootstrapping things. I still like to upstream it, but got distracted by many other posts.

        Show
        oflebbe Olaf Flebbe added a comment - The system I presented in Budapest is doing this. http://github.com/oflebbe/inst But the system is outdated now, since Michael found a better way to use puppet for bootstrapping things. I still like to upstream it, but got distracted by many other posts.
        Hide
        rvs Roman Shaposhnik added a comment -

        Olaf Flebbe agreed that it is time for us to do native systemd support (although, truth be told I still can't seem to bring myself to care about it – may be Ubuntu 16.04 is going to change that for me). FWIW I actually filed similar kind of JIRA BIGTOP-116 back when systemd just first came out. Feel free to close either one

        Show
        rvs Roman Shaposhnik added a comment - Olaf Flebbe agreed that it is time for us to do native systemd support (although, truth be told I still can't seem to bring myself to care about it – may be Ubuntu 16.04 is going to change that for me). FWIW I actually filed similar kind of JIRA BIGTOP-116 back when systemd just first came out. Feel free to close either one
        Hide
        oflebbe Olaf Flebbe added a comment - - edited

        For systemd units one would have to reuse hadoop-daemon.sh and modify it a bit.
        After thinking about it I decided to add the pidfile: to the template and this should fix it for now. (surprisingly zookeeper already has it).

        Show
        oflebbe Olaf Flebbe added a comment - - edited For systemd units one would have to reuse hadoop-daemon.sh and modify it a bit. After thinking about it I decided to add the pidfile: to the template and this should fix it for now. (surprisingly zookeeper already has it).
        Hide
        oflebbe Olaf Flebbe added a comment -

        committed

        Show
        oflebbe Olaf Flebbe added a comment - committed

          People

          • Assignee:
            oflebbe Olaf Flebbe
            Reporter:
            oflebbe Olaf Flebbe
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development