[MESOS-2369] Segfault when mesos-slave tries to clean up docker containers on startup

• Type: Bug
• Status: Accepted
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.21.1, 1.2.0, 1.3.0
• Component/s: docker

Environment

Debian Jessie, mesos package 0.21.1-1.2.debian77
docker 1.3.2 build 39fa2fa

• Assignee: Anand Mazumdar
• Reporter: Pas
• Created: 18/Feb/15 11:22
• Updated: 13/Feb/17 18:11

Description

I did a gdb backtrace, it seems like a stack overflow due to a bit too much recursion.

The interesting aspect is that after running mesos-slave with strace -f -b execve it successfully proceeded with the docker cleanup. However, there were a few strace sessions (on other slaves) where I was able to observe the SIGSEGV, and it was around (or a bit before) the "docker ps -a" call, because docker got a broken pipe shortly, then got killed by the propagating SIGSEGV signal.

....

#59296 0x00007ffff6e7cd98 in process::Future<std::string> process::Future<unsigned long>::then<std::string>(std::tr1::function<process::Future<std::string> (unsigned long const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59297 0x00007ffff6e4f5d3 in process::io::internal::_read(int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59298 0x00007ffff6e5012c in process::io::internal::__read(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59299 0x00007ffff6e53000 in std::tr1::_Function_handler<process::Future<std::string> (unsigned long const&), std::tr1::_Bind<process::Future<std::string> (*(std::tr1::_Placeholder<1>, int, std::tr1::shared_ptr<std::string>, boost::shared_array<char>, unsigned long))(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long)> >::_M_invoke(std::tr1::_Any_data const&, unsigned long const&) () from /usr/local/lib/libmesos-0.21.1.so
#59300 0x00007ffff6e7d23b in void process::internal::thenf<unsigned long, std::string>(std::tr1::shared_ptr<process::Promise<std::string> > const&, std::tr1::function<process::Future<std::string> (unsigned long const&)> const&, process::Future<unsigned long> const&) ()
   from /usr/local/lib/libmesos-0.21.1.so
#59301 0x00007ffff689ee60 in process::Future<unsigned long>::onAny(std::tr1::function<void (process::Future<unsigned long> const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59302 0x00007ffff6e7cd98 in process::Future<std::string> process::Future<unsigned long>::then<std::string>(std::tr1::function<process::Future<std::string> (unsigned long const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59303 0x00007ffff6e4f5d3 in process::io::internal::_read(int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59304 0x00007ffff6e5012c in process::io::internal::__read(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59305 0x00007ffff6e53000 in std::tr1::_Function_handler<process::Future<std::string> (unsigned long const&), std::tr1::_Bind<process::Future<std::string> (*(std::tr1::_Placeholder<1>, int, std::tr1::shared_ptr<std::string>, boost::shared_array<char>, unsigned long))(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long)> >::_M_invoke(std::tr1::_Any_data const&, unsigned long const&) () from /usr/local/lib/libmesos-0.21.1.so
#59306 0x00007ffff6e7d23b in void process::internal::thenf<unsigned long, std::string>(std::tr1::shared_ptr<process::Promise<std::string> > const&, std::tr1::function<process::Future<std::string> (unsigned long const&)> const&, process::Future<unsigned long> const&) ()
   from /usr/local/lib/libmesos-0.21.1.so
#59307 0x00007ffff689ee60 in process::Future<unsigned long>::onAny(std::tr1::function<void (process::Future<unsigned long> const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59308 0x00007ffff6e7cd98 in process::Future<std::string> process::Future<unsigned long>::then<std::string>(std::tr1::function<process::Future<std::string> (unsigned long const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59309 0x00007ffff6e4f5d3 in process::io::internal::_read(int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59310 0x00007ffff6e5012c in process::io::internal::__read(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59311 0x00007ffff6e53000 in std::tr1::_Function_handler<process::Future<std::string> (unsigned long const&), std::tr1::_Bind<process::Future<std::string> (*(std::tr1::_Placeholder<1>, int, std::tr1::shared_ptr<std::string>, boost::shared_array<char>, unsigned long))(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long)> >::_M_invoke(std::tr1::_Any_data const&, unsigned long const&) () from /usr/local/lib/libmesos-0.21.1.so
#59312 0x00007ffff6e7d23b in void process::internal::thenf<unsigned long, std::string>(std::tr1::shared_ptr<process::Promise<std::string> > const&, std::tr1::function<process::Future<std::string> (unsigned long const&)> const&, process::Future<unsigned long> const&) ()
   from /usr/local/lib/libmesos-0.21.1.so
#59313 0x00007ffff689ee60 in process::Future<unsigned long>::onAny(std::tr1::function<void (process::Future<unsigned long> const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59314 0x00007ffff6e7cd98 in process::Future<std::string> process::Future<unsigned long>::then<std::string>(std::tr1::function<process::Future<std::string> (unsigned long const&)> const&) const () from /usr/local/lib/libmesos-0.21.1.so
#59315 0x00007ffff6e4f5d3 in process::io::internal::_read(int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59316 0x00007ffff6e5012c in process::io::internal::__read(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long) () from /usr/local/lib/libmesos-0.21.1.so
#59317 0x00007ffff6e53000 in std::tr1::_Function_handler<process::Future<std::string> (unsigned long const&), std::tr1::_Bind<process::Future<std::string> (*(std::tr1::_Placeholder<1>, int, std::tr1::shared_ptr<std::string>, boost::shared_array<char>, unsigned long))(unsigned long, int, std::tr1::shared_ptr<std::string> const&, boost::shared_array<char> const&, unsigned long)> >::_M_invoke(std::tr1::_Any_data const&, unsigned long const&) () from /usr/local/lib/libmesos-0.21.1.so
#59318 0x00007ffff6e7d23b in void process::internal::thenf<unsigned long, std::string>(std::tr1::shared_ptr<process::Promise<std::string> > const&, std::tr1::function<process::Future<std::string> (unsigned long const&)> const&, process::Future<unsigned long> const&) ()
   from /usr/local/lib/libmesos-0.21.1.so
#59319 0x00007ffff6c0f138 in process::Future<unsigned long>::set(unsigned long const&) () from /usr/local/lib/libmesos-0.21.1.so
#59320 0x00007ffff6e46a64 in process::io::internal::read(int, void*, unsigned long, std::tr1::shared_ptr<process::Promise<unsigned long> > const&, process::Future<short> const&) () from /usr/local/lib/libmesos-0.21.1.so
#59321 0x00007ffff6e59508 in process::Future<short>::set(short const&) () from /usr/local/lib/libmesos-0.21.1.so
#59322 0x00007ffff6e59469 in process::Future<short>::set(short const&) () from /usr/local/lib/libmesos-0.21.1.so
#59323 0x00007ffff6e3422e in process::polled(ev_loop*, ev_io*, int) () from /usr/local/lib/libmesos-0.21.1.so
#59324 0x00007ffff6ead365 in ev_invoke_pending (loop=0x7ffff7ddb460 <default_loop_struct>) at ev.c:2994
#59325 0x00007ffff6eb03c5 in ev_run (loop=0x7ffff7ddb460 <default_loop_struct>, flags=<optimized out>) at ev.c:3394
#59326 0x00007ffff6e3235b in process::serve(void*) () from /usr/local/lib/libmesos-0.21.1.so
#59327 0x00007ffff474d0a4 in start_thread (arg=0x7fffebe27700) at pthread_create.c:309
#59328 0x00007ffff4481ccd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Issue Links

• relates to: MESOS-7102 Crash when sending a SIGUSR1 signal to the agent.