Qpid
  1. Qpid
  2. QPID-3147

Misconfigured tracing/logging can lead to hung threads in logging stack

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.10
    • Fix Version/s: 0.10
    • Component/s: C++ Client
    • Environment:

      Fedora 14 32-bit

      Description

      Given the following trace and logging environment variable mix-up like this:

      QPID_TRACE=debug+
      QPID_LOG_ENABLE=

      A C++ client that embeds the Qpid runtime can get hung although it has recognized the misconfiguration.

      #0 0x00555416 in __kernel_vsyscall ()
      #1 0x00f49367 in syscall () at ../sysdeps/unix/sysv/linux/i386/syscall.S:30
      #2 0x002bbfca in _cxxabiv1::_cxa_guard_acquire (g=0x2171ea0)
      at ../../../../libstdc+-v3/libsupc+/guard.cc:293
      #3 0x020ca3b8 in instance ()
      at /usr/include/boost/pool/detail/singleton.hpp:83
      #4 qpid::log::Logger::instance () at qpid/log/Logger.cpp:47
      #5 0x020cd324 in qpid::log::Statement::Initializer::Initializer (
      this=0x2171b88, s=...) at qpid/log/Statement.cpp:61
      #6 0x02075b6c in qpid::Exception::Exception (this=0x880b370, msg=
      "Error in environment variables: in option 'trace': invalid bool value\n")
      at qpid/Exception.cpp:31
      #7 0x0208095d in Exception (this=0xbf8d0df0, argc=0, argv=0x0, configFile="",
      allowUnknown=false) at ../include/qpid/Options.h:210
      #8 qpid::Options::parse (this=0xbf8d0df0, argc=0, argv=0x0, configFile="",
      allowUnknown=false) at qpid/Options.cpp:352
      #9 0x020c9c57 in qpid::log::Logger::Logger (this=0x2171ec0)
      at qpid/log/Logger.cpp:55
      #10 0x020ca330 in instance ()
      at /usr/include/boost/pool/detail/singleton.hpp:83
      #11 object_creator () at /usr/include/boost/pool/detail/singleton.hpp:66
      #12 __static_initialization_and_destruction_0 ()
      at /usr/include/boost/pool/detail/singleton.hpp:95
      #13 global constructors keyed to Logger.cpp(void) () at qpid/log/Logger.cpp:156
      #14 0x020ebc6d in __do_global_ctors_aux () from /usr/lib/libqpidcommon.so.2
      #15 0x01fc2a84 in _init () from /usr/lib/libqpidcommon.so.2
      #16 0x004f68fc in call_init (l=<value optimized out>, argc=3, argv=0xbf8d1634,
      env=0x87feb58) at dl-init.c:68
      #17 0x004f6a19 in _dl_init (main_map=<value optimized out>, argc=3,
      argv=0xbf8d1634, env=0x87feb58) at dl-init.c:132
      #18 0x004fa74f in dl_open_worker (a=0xbf8d10d0) at dl-open.c:464
      #19 0x004f6786 in _dl_catch_error (objname=0xbf8d10f8, errstring=0xbf8d10f4,
      mallocedp=0xbf8d10ff, operate=0x4fa3b0 <dl_open_worker>, args=0xbf8d10d0)
      at dl-error.c:178

        Activity

        Hide
        Alan Conway added a comment - - edited

        The problem here is that the code which parses logger options itself uses the logger (via qpid::Exception constructor) if there is invalid configuration. This results in a recursive call to the instance() function of the boost singleton code we are using, which deadlocks due to a lock generated by gcc, presumably to protect the static local variable from concurrent access.

        Show
        Alan Conway added a comment - - edited The problem here is that the code which parses logger options itself uses the logger (via qpid::Exception constructor) if there is invalid configuration. This results in a recursive call to the instance() function of the boost singleton code we are using, which deadlocks due to a lock generated by gcc, presumably to protect the static local variable from concurrent access.
        Hide
        Alan Conway added a comment -

        Fixed on trunk r1083884. Candidate for 0.10 branch

        QPID-3147: Misconfigured tracing/logging can lead to hung threads in logging stack

        The hang was caused by re-entrant attempts to initialize the Logger
        singleton when an exception was thrown during logger configuration.
        The fix is to disable exception logging temporarily while the logger
        is constructed.

        Show
        Alan Conway added a comment - Fixed on trunk r1083884. Candidate for 0.10 branch QPID-3147 : Misconfigured tracing/logging can lead to hung threads in logging stack The hang was caused by re-entrant attempts to initialize the Logger singleton when an exception was thrown during logger configuration. The fix is to disable exception logging temporarily while the logger is constructed.
        Hide
        michael j. goulish added a comment -

        nice fix. seems like zero risk.
        nihil obstat. imprimatur.

        Show
        michael j. goulish added a comment - nice fix. seems like zero risk. nihil obstat. imprimatur.
        Hide
        Alan Conway added a comment -

        Committed 0.10 branch r1083897

        Show
        Alan Conway added a comment - Committed 0.10 branch r1083897

          People

          • Assignee:
            Alan Conway
            Reporter:
            Pete MacKinnon
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development