ZooKeeper
  1. ZooKeeper
  2. ZOOKEEPER-981

Hang in zookeeper_close() in the multi-threaded C client

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 3.3.2
    • Fix Version/s: 3.4.0
    • Component/s: c client
    • Labels:
      None
    • Environment:

      Debian Squeeze, Linux 2.6.32-5, x86_64

    • Hadoop Flags:
      Reviewed

      Description

      I saw a hang once when my C++ application called the zookeeper_close() method of the multi-threaded Zookeeper client library. The stack trace of the hung thread was the following:

      Thread 8 (Thread 5644):
      #0 0x00007f5d7bb5bbe4 in __lll_lock_wait () from /lib/libpthread.so.0
      #1 0x00007f5d7bb59ad0 in pthread_cond_broadcast@@GLIBC_2.3.2 () from /lib/libpthread.so.0
      #2 0x00007f5d793628f6 in unlock_completion_list (l=0x32b4d68) at .../zookeeper/src/c/src/mt_adaptor.c:66
      #3 0x00007f5d79354d4b in free_completions (zh=0x32b4c80, callCompletion=1, reason=-116) at .../zookeeper/src/c/src/zookeeper.c:1069
      #4 0x00007f5d79355008 in cleanup_bufs (zh=0x32b4c80, callCompletion=1, rc=-116) at .../thirdparty/zookeeper/src/c/src/zookeeper.c:1125
      #5 0x00007f5d79353200 in destroy (zh=0x32b4c80) at .../thirdparty/zookeeper/src/c/src/zookeeper.c:366
      #6 0x00007f5d79358e0e in zookeeper_close (zh=0x32b4c80) at .../zookeeper/src/c/src/zookeeper.c:2326
      #7 0x00007f5d79356d18 in api_epilog (zh=0x32b4c80, rc=0) at .../zookeeper/src/c/src/zookeeper.c:1661
      #8 0x00007f5d79362f2f in adaptor_finish (zh=0x32b4c80) at .../zookeeper/src/c/src/mt_adaptor.c:205
      #9 0x00007f5d79358c8c in zookeeper_close (zh=0x32b4c80) at .../zookeeper/src/c/src/zookeeper.c:2297
      ...

      The omitted part of the stack trace is entirely within my application, and contains no other calls to/from the Zookeeper client. In particular, I am not calling zookeeper_close() from within a completion handler or any of the library's threads.

      I haven't been able to reproduce this, and when I encountered this I wasn't capturing logging from the client library, so unfortunately I don't have any more information at this time. But I will update this JIRA if I see it again.

      1. ZOOKEEPER-981-v1.patch
        1 kB
        Mahadev konar
      2. ZOOKEEPER-981.tar.gz
        1 kB
        tsulin
      3. zookeeper-981.patch
        1 kB
        Jeremy Stribling

        Activity

        Jeremy Stribling created issue -
        Patrick Hunt made changes -
        Field Original Value New Value
        Fix Version/s 3.3.3 [ 12315482 ]
        Fix Version/s 3.4.0 [ 12314469 ]
        Priority Minor [ 4 ] Critical [ 2 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471408 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471408 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471410 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471410 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471411 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tar.gz [ 12471412 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tar.gz [ 12471412 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tgz [ 12471411 ]
        tsulin made changes -
        Attachment ZOOKEEPER-981.tar.gz [ 12471413 ]
        Benjamin Reed made changes -
        Fix Version/s 3.3.3 [ 12315482 ]
        Jeremy Stribling made changes -
        Attachment zookeeper-981.patch [ 12472730 ]
        Mahadev konar made changes -
        Attachment ZOOKEEPER-981-v1.patch [ 12486732 ]
        Mahadev konar made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Patrick Hunt made changes -
        Assignee Jeremy Stribling [ strib ]
        Jeremy Stribling made changes -
        Assignee Jeremy Stribling [ strib ] Patrick Hunt [ phunt ]
        Mahadev konar made changes -
        Assignee Patrick Hunt [ phunt ] Jeremy Stribling [ strib ]
        Mahadev konar made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Mahadev konar made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jeremy Stribling
            Reporter:
            Jeremy Stribling
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development