[MESOS-3201] Libev handle_async can deadlock with run_in_event_loop

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.24.0
• Component/s: libprocess
• Labels: libprocess mesosphere
• Sprint: Mesosphere Sprint 16
• Story Points: 3
• Assignee: Joris Van Remoortere
• Reporter: Joris Van Remoortere
• Shepherd: Benjamin Hindman
• Created: 04/Aug/15 17:58
• Updated: 17/Aug/15 23:19
• Resolved: 17/Aug/15 23:19

Description

Due to the arbitrary nature of the functions that are executed in handle_async, invoking them under the (A) watchers_mutex can lead to deadlocks if (B) is acquired before calling run_in_event_loop and (B) is also acquired within the arbitrary function.

==82679== Thread #10: lock order "0x60774F8 before 0x60768C0" violated
==82679== 
==82679== Observed (incorrect) order is: acquisition of lock at 0x60768C0
==82679==    at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==82679==    by 0x692C9B: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748)
==82679==    by 0x6950BF: std::mutex::lock() (mutex:134)
==82679==    by 0x696219: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*)::{lambda(std::mutex*)#1}::operator()(std::mutex*) const (synchronized.hpp:58)
==82679==    by 0x696238: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*)::{lambda(std::mutex*)#1}::_FUN(std::mutex*) (synchronized.hpp:58)
==82679==    by 0x6984CF: Synchronized<std::mutex>::Synchronized(std::mutex*, void (*)(std::mutex*), void (*)(std::mutex*)) (synchronized.hpp:35)
==82679==    by 0x6962DE: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*) (synchronized.hpp:60)
==82679==    by 0x728FE1: process::handle_async(ev_loop*, ev_async*, int) (libev.cpp:48)
==82679==    by 0x761384: ev_invoke_pending (ev.c:2994)
==82679==    by 0x7643C4: ev_run (ev.c:3394)
==82679==    by 0x728E37: ev_loop (ev.h:826)
==82679==    by 0x729469: process::EventLoop::run() (libev.cpp:135)
==82679== 
==82679==  followed by a later acquisition of lock at 0x60774F8
==82679==    at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==82679==    by 0x4C6F9D: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748)
==82679==    by 0x4C6FED: __gthread_recursive_mutex_lock(pthread_mutex_t*) (gthr-default.h:810)
==82679==    by 0x4F5D3D: std::recursive_mutex::lock() (mutex:175)
==82679==    by 0x516513: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::operator()(std::recursive_mutex*) const (synchronized.hpp:58)
==82679==    by 0x516532: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::_FUN(std::recursive_mutex*) (synchronized.hpp:58)
==82679==    by 0x52E619: Synchronized<std::recursive_mutex>::Synchronized(std::recursive_mutex*, void (*)(std::recursive_mutex*), void (*)(std::recursive_mutex*)) (synchronized.hpp:35)
==82679==    by 0x5165D4: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*) (synchronized.hpp:60)
==82679==    by 0x6BF4E1: process::ProcessManager::use(process::UPID const&) (process.cpp:2127)
==82679==    by 0x6C2B8C: process::ProcessManager::terminate(process::UPID const&, bool, process::ProcessBase*) (process.cpp:2604)
==82679==    by 0x6C6C3C: process::terminate(process::UPID const&, bool) (process.cpp:3107)
==82679==    by 0x692B65: process::Latch::trigger() (latch.cpp:53)

This was introduced in https://github.com/apache/mesos/commit/849fc4d361e40062073324153ba97e98e294fdf2