Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3201

Libev handle_async can deadlock with run_in_event_loop

    XMLWordPrintableJSON

    Details

    • Sprint:
      Mesosphere Sprint 16
    • Story Points:
      3

      Description

      Due to the arbitrary nature of the functions that are executed in handle_async, invoking them under the (A) watchers_mutex can lead to deadlocks if (B) is acquired before calling run_in_event_loop and (B) is also acquired within the arbitrary function.

      ==82679== Thread #10: lock order "0x60774F8 before 0x60768C0" violated
      ==82679== 
      ==82679== Observed (incorrect) order is: acquisition of lock at 0x60768C0
      ==82679==    at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
      ==82679==    by 0x692C9B: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748)
      ==82679==    by 0x6950BF: std::mutex::lock() (mutex:134)
      ==82679==    by 0x696219: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*)::{lambda(std::mutex*)#1}::operator()(std::mutex*) const (synchronized.hpp:58)
      ==82679==    by 0x696238: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*)::{lambda(std::mutex*)#1}::_FUN(std::mutex*) (synchronized.hpp:58)
      ==82679==    by 0x6984CF: Synchronized<std::mutex>::Synchronized(std::mutex*, void (*)(std::mutex*), void (*)(std::mutex*)) (synchronized.hpp:35)
      ==82679==    by 0x6962DE: Synchronized<std::mutex> synchronize<std::mutex>(std::mutex*) (synchronized.hpp:60)
      ==82679==    by 0x728FE1: process::handle_async(ev_loop*, ev_async*, int) (libev.cpp:48)
      ==82679==    by 0x761384: ev_invoke_pending (ev.c:2994)
      ==82679==    by 0x7643C4: ev_run (ev.c:3394)
      ==82679==    by 0x728E37: ev_loop (ev.h:826)
      ==82679==    by 0x729469: process::EventLoop::run() (libev.cpp:135)
      ==82679== 
      ==82679==  followed by a later acquisition of lock at 0x60774F8
      ==82679==    at 0x4C32145: pthread_mutex_lock (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
      ==82679==    by 0x4C6F9D: __gthread_mutex_lock(pthread_mutex_t*) (gthr-default.h:748)
      ==82679==    by 0x4C6FED: __gthread_recursive_mutex_lock(pthread_mutex_t*) (gthr-default.h:810)
      ==82679==    by 0x4F5D3D: std::recursive_mutex::lock() (mutex:175)
      ==82679==    by 0x516513: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::operator()(std::recursive_mutex*) const (synchronized.hpp:58)
      ==82679==    by 0x516532: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*)::{lambda(std::recursive_mutex*)#1}::_FUN(std::recursive_mutex*) (synchronized.hpp:58)
      ==82679==    by 0x52E619: Synchronized<std::recursive_mutex>::Synchronized(std::recursive_mutex*, void (*)(std::recursive_mutex*), void (*)(std::recursive_mutex*)) (synchronized.hpp:35)
      ==82679==    by 0x5165D4: Synchronized<std::recursive_mutex> synchronize<std::recursive_mutex>(std::recursive_mutex*) (synchronized.hpp:60)
      ==82679==    by 0x6BF4E1: process::ProcessManager::use(process::UPID const&) (process.cpp:2127)
      ==82679==    by 0x6C2B8C: process::ProcessManager::terminate(process::UPID const&, bool, process::ProcessBase*) (process.cpp:2604)
      ==82679==    by 0x6C6C3C: process::terminate(process::UPID const&, bool) (process.cpp:3107)
      ==82679==    by 0x692B65: process::Latch::trigger() (latch.cpp:53)
      

      This was introduced in https://github.com/apache/mesos/commit/849fc4d361e40062073324153ba97e98e294fdf2

        Attachments

          Activity

            People

            • Assignee:
              jvanremoortere Joris Van Remoortere
              Reporter:
              jvanremoortere Joris Van Remoortere
              Shepherd:
              Benjamin Hindman
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: