Qpid
  1. Qpid
  2. QPID-2631

Race in qpid::client::Bounds causes (rare) deadlock

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.5, 0.6
    • Fix Version/s: 0.7
    • Component/s: C++ Client
    • Labels:
      None

      Description

      There is a race condition in the use of Bounds in SessionImpl::sendFrame. This function sends the frame first, then calls
      Bounds::expand(). But it's possible the network thread calls Bounds::reduce() between sending the frame and calling expand. If the Bounds::current value is 0
      that reduce() is lost. If enough reduce() calls are lost in this way eventually we will deadlock.

      In investigating this it also became clear that the connection frames weren't correctly accounted for (i.e. the bounds are never expended for connection frames, though they are included in the byte count passed in on reduce()). Though this shouldn't actually cause any problem it is logically incorrect, unintuitive and could mask problems that are hard to diagnose.

        Activity

        Hide
        zhangyb added a comment - - edited

        This bug seems to have been fixed in the o.8 version through the status of the problem.But I still found the bug in the 0.14,why?
        #0 0x0000003d55e0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
        #1 0x00007f4610584675 in wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>)
        at ../include/qpid/sys/posix/Condition.h:63
        #2 wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at ../include/qpid/sys/Monitor.h:41
        #3 wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at ./qpid/sys/Waitable.h:91
        #4 qpid::client::Bounds::expand (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at qpid/client/Bounds.cpp:39
        #5 0x00007f46105b05a0 in qpid::client::SessionImpl::sendFrame (this=0x1ec6e60, frame=..., canBlock=true)
        at qpid/client/SessionImpl.cpp:514
        #6 0x00007f46105b23b7 in qpid::client::SessionImpl::sendContent (this=0x1ec6e60, content=...) at qpid/client/SessionImpl.cpp:429
        #7 0x00007f46105b4641 in qpid::client::SessionImpl::sendCommand (this=0x1ec6e60, command=..., content=0x7f45c4031c90)
        at qpid/client/SessionImpl.cpp:399
        #8 0x00007f46105b4839 in qpid::client::SessionImpl::send (this=<value optimized out>, command=<value optimized out>,
        content=<value optimized out>) at qpid/client/SessionImpl.cpp:307

        #9 0x00007f4610576d3d in qpid::client::no_keyword::AsyncSession_0_10::messageTransfer (this=0x1ec7da8,
        destination=<value optimized out>, acceptMode=<value optimized out>, acquireMode=<value optimized out>, content=...,
        sync=false) at ./qpid/client/no_keyword/AsyncSession_0_10.cpp:63

        Show
        zhangyb added a comment - - edited This bug seems to have been fixed in the o.8 version through the status of the problem.But I still found the bug in the 0.14,why? #0 0x0000003d55e0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4610584675 in wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at ../include/qpid/sys/posix/Condition.h:63 #2 wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at ../include/qpid/sys/Monitor.h:41 #3 wait (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at ./qpid/sys/Waitable.h:91 #4 qpid::client::Bounds::expand (this=0x1e8a5f0, sizeRequired=1080, block=<value optimized out>) at qpid/client/Bounds.cpp:39 #5 0x00007f46105b05a0 in qpid::client::SessionImpl::sendFrame (this=0x1ec6e60, frame=..., canBlock=true) at qpid/client/SessionImpl.cpp:514 #6 0x00007f46105b23b7 in qpid::client::SessionImpl::sendContent (this=0x1ec6e60, content=...) at qpid/client/SessionImpl.cpp:429 #7 0x00007f46105b4641 in qpid::client::SessionImpl::sendCommand (this=0x1ec6e60, command=..., content=0x7f45c4031c90) at qpid/client/SessionImpl.cpp:399 #8 0x00007f46105b4839 in qpid::client::SessionImpl::send (this=<value optimized out>, command=<value optimized out>, content=<value optimized out>) at qpid/client/SessionImpl.cpp:307 #9 0x00007f4610576d3d in qpid::client::no_keyword::AsyncSession_0_10::messageTransfer (this=0x1ec7da8, destination=<value optimized out>, acceptMode=<value optimized out>, acquireMode=<value optimized out>, content=..., sync=false) at ./qpid/client/no_keyword/AsyncSession_0_10.cpp:63
        Hide
        Gordon Sim added a comment -

        The bug as described is quite narrow and specific and was fixed by http://svn.apache.org/viewvc?view=revision&revision=948936. The backtrace above doesn't imply the same issue. It is to some degree expected that clients may block at this point (while waiting for existing data to be written to the socket). However they should not block indefinitely. If you believe you are getting into an erroneous state, i.e. a wait that never clears, please raise a new bug and/or describe in more detail what you observe.

        Show
        Gordon Sim added a comment - The bug as described is quite narrow and specific and was fixed by http://svn.apache.org/viewvc?view=revision&revision=948936 . The backtrace above doesn't imply the same issue. It is to some degree expected that clients may block at this point (while waiting for existing data to be written to the socket). However they should not block indefinitely. If you believe you are getting into an erroneous state, i.e. a wait that never clears, please raise a new bug and/or describe in more detail what you observe.

          People

          • Assignee:
            Gordon Sim
            Reporter:
            Gordon Sim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development