Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-2668

Slave fails to recover when there are still processes left in its cgroup

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.22.1
    • 0.22.1, 0.23.0
    • None
    • None

    • 0.22.1-rc5
      CoreOS

    Description

      mesos-slave process enters failed state with "A slave (or child process) is still running, please check the process(es) '{ ... }' listed in /sys/fs/cgroup/cpu,cpuacct/mesos/slave/cgroups.proc"

      Shortly after starting a test cluster we are seeing mesos-slaves enter a permanent failed state after a short period (~10 minutes or so). Here is the failure we are seeing with mesos-slave:

      Apr 27 23:29:33 ip-10-229-44-239.ec2.internal systemd[1]: mesos-slave.service: main process exited, code=exited, status=1/FAILURE
Apr 27 23:29:33 ip-10-229-44-239.ec2.internal systemd[1]: Unit mesos-slave.service entered failed state.
Apr 27 23:29:33 ip-10-229-44-239.ec2.internal systemd[1]: mesos-slave.service failed.
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal systemd[1]: mesos-slave.service holdoff time over, scheduling restart.
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal systemd[1]: Stopping Mesos Slave...
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal systemd[1]: Starting Mesos Slave...
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal ping[9272]: PING leader.mesos (10.155.13.144) 56(84) bytes of data.
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal ping[9272]: 64 bytes from ip-10-155-13-144.ec2.internal (10.155.13.144): icmp_seq=1 ttl=49 time=0.277 ms
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal ping[9272]: --- leader.mesos ping statistics ---
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal ping[9272]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal ping[9272]: rtt min/avg/max/mdev = 0.277/0.277/0.277/0.000 ms
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal systemd[1]: Started Mesos Slave.
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:38.908716  9274 logging.cpp:172] INFO level logging started!
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:38.909025  9274 main.cpp:156] Build: 2015-04-25 01:51:59 by
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:38.909044  9274 main.cpp:158] Version: 0.22.1
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:38.909054  9274 main.cpp:161] Git tag: 0.22.1-rc5
Apr 27 23:29:38 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:38.909065  9274 main.cpp:165] Git SHA: 13e0536a4522c5674abc920ee9b8597d83c5352a
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:39.012763  9274 containerizer.cpp:110] Using isolation: cgroups/cpu,cgroups/mem
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:39.027866  9274 linux_launcher.cpp:94] Using /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux launcher
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:39.028177  9274 main.cpp:200] Starting Mesos slave
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@716: Client environment:host.name=ip-10-229-44-239.ec2.internal
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@724: Client environment:os.arch=3.19.0
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Fri Mar 6 00:23:51 UTC 2015
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@741: Client environment:user.home=/root
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@log_env@753: Client environment:user.dir=/
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,028:9274(0x7f8390836700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=leader.mesos:2181 sessionTimeout=10000 watcher=0x7f8393fa9ae0 sessionId=0 sessionPasswd=<null> context=0x7f8364000970 flags=0
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:39.028657  9274 slave.cpp:174] Slave started on 1)@10.229.44.239:5051
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: I0427 23:29:39.028688  9274 slave.cpp:194] Moving slave process into its own cgroup for subsystem: cpu
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,030:9274(0x7f83877c8700):ZOO_INFO@check_events@1703: initiated connection to server [10.155.13.144:2181]
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: A slave (or child process) is still running, please check the process(es) '{ 2242, 2243, 2244 }' listed in /sys/fs/cgroup/cpu,cpuacct/mesos/slave/cgroups.proc
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal mesos-slave[9274]: 2015-04-27 23:29:39,032:9274(0x7f83877c8700):ZOO_INFO@check_events@1750: session establishment complete on server [10.155.13.144:2181], sessionId=0x274cfd1112750268, negotiated timeout=10000
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal systemd[1]: mesos-slave.service: main process exited, code=exited, status=1/FAILURE
Apr 27 23:29:39 ip-10-229-44-239.ec2.internal systemd[1]: Unit mesos-slave.service entered failed state.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MESOS-2554 Slave flaps when using --slave_subsystems that are not used for isolation.

          • Major - Major loss of function.
          • Accepted

          Improvement - An improvement or enhancement to an existing feature or task. MESOS-2461 Slave should provide details on processes running in its cgroups

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. MESOS-2462 Add option for Subprocess to set a death signal for the forked child

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              tnachen Timothy Chen
              jlingmann Jeremy Lingmann
              Benjamin Hindman Benjamin Hindman
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: