Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-676

50%-75% connection loss exceptions using zkpython

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.2.2
    • None
    • contrib, contrib-bindings
    • None
    • Mac OS X 10.5.8, MacBook Air Intel Core 2 Duo @ 1.86GHz, Python 2.5.1, ZooKeeper 3.2.2 Standalone

    Description

      I get about 50-75% connection loss exceptions and about 10% Bus Error when using the contrib/zkpython zookeeper.so. Below is the exception:

      2010-02-21 16:57:56,138:18481(0xb0081000):ZOO_ERROR@handle_socket_error_msg@1359: Socket [fe80::1002:885:7f00:1:2181] zk retcode=-4, errno=47(Address family not supported by protocol family): connect() call failed
      Traceback (most recent call last):
      File "./zksh.py", line 63, in <module>
      2010-02-21 16:57:56,138:18481(0xb0081000):ZOO_INFO@check_events@1439: initiated connection to server [127.0.0.1:2181]
      zkcli.dispatch(cmd,*args)
      File "./zksh.py", line 56, in dispatch
      returned = run(*args)
      File "./zksh.py", line 48, in ls
      print "\n".join(self.cmd.listNode(node))
      File "/Users/josh/git/zktools/commands.py", line 22, in listNode
      for path in zookeeper.get_children(self.zk, node):
      zookeeper.ConnectionLossException: connection loss

      I've run this in gdb and have this backtrace:

      #0 free_pywatcher (pw=0x0) at src/c/zookeeper.c:199
      #1 0x0025ae09 in pyzoo_exists (self=0x0, args=0x0) at src/c/zookeeper.c:765
      #2 0x0018f51e in PyEval_EvalFrameEx ()
      #3 0x00191173 in PyEval_EvalCodeEx ()
      #4 0x0013b488 in PyFunction_SetClosure ()
      #5 0x00121505 in PyObject_Call ()
      #6 0x0018fcd0 in PyEval_EvalFrameEx ()
      #7 0x00191173 in PyEval_EvalCodeEx ()
      #8 0x0013b488 in PyFunction_SetClosure ()
      #9 0x00121505 in PyObject_Call ()
      #10 0x0018fcd0 in PyEval_EvalFrameEx ()
      #11 0x00191173 in PyEval_EvalCodeEx ()
      #12 0x0018f79d in PyEval_EvalFrameEx ()
      #13 0x00191173 in PyEval_EvalCodeEx ()
      #14 0x00191260 in PyEval_EvalCode ()
      #15 0x001a883c in PyErr_Display ()
      #16 0x001aa4ab in PyRun_InteractiveOneFlags ()
      #17 0x001aa5f9 in PyRun_InteractiveLoopFlags ()
      #18 0x001aaa2b in PyRun_AnyFileExFlags ()
      #19 0x001b5a57 in Py_Main ()
      #20 0x00001fca in ?? ()

      zookeeper.c @ line 199:

      void free_pywatcher( pywatcher_t *pw)
      {
      Py_DECREF(pw->callback);
      free(pw);
      }

      That's as far as I've dug so far – I ended up just writing a retry decorator to get around it for now. On the same machine, the zkCli.sh test client works flawlessly. Also, here's the Mac OS X Bus Error trace:

      Process: Python [18556]
      Path: /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
      Identifier: Python
      Version: ??? (???)
      Code Type: X86 (Native)
      Parent Process: bash [18436]

      Interval Since Last Report: 3323078 sec
      Crashes Since Last Report: 50
      Per-App Interval Since Last Report: 0 sec
      Per-App Crashes Since Last Report: 38

      Date/Time: 2010-02-21 17:07:27.399 -0800
      OS Version: Mac OS X 10.5.8 (9L31a)
      Report Version: 6
      Anonymous UUID: FA533BDA-50B2-47A9-931C-6F2614C741F0

      Exception Type: EXC_BAD_ACCESS (SIGBUS)
      Exception Codes: KERN_PROTECTION_FAILURE at 0x0000000000000004
      Crashed Thread: 0

      Thread 0 Crashed:
      0 zookeeper.so 0x002332bd free_pywatcher + 10 (zookeeper.c:199)
      1 zookeeper.so 0x00239e09 pyzoo_exists + 984 (zookeeper.c:765)
      2 org.python.python 0x0018f51e PyEval_EvalFrameEx + 17116
      3 org.python.python 0x0018f700 PyEval_EvalFrameEx + 17598
      4 org.python.python 0x00191173 PyEval_EvalCodeEx + 1638
      5 org.python.python 0x0013b488 PyFunction_SetClosure + 2667
      6 org.python.python 0x00121505 PyObject_Call + 50
      7 org.python.python 0x0018fcd0 PyEval_EvalFrameEx + 19086
      8 org.python.python 0x00191173 PyEval_EvalCodeEx + 1638
      9 org.python.python 0x0013b488 PyFunction_SetClosure + 2667
      10 org.python.python 0x00121505 PyObject_Call + 50
      11 org.python.python 0x0018fcd0 PyEval_EvalFrameEx + 19086
      12 org.python.python 0x00191173 PyEval_EvalCodeEx + 1638
      13 org.python.python 0x00191260 PyEval_EvalCode + 87
      14 org.python.python 0x001a883c PyErr_Display + 1896
      15 org.python.python 0x001a8e66 PyRun_FileExFlags + 135
      16 org.python.python 0x001aa7d2 PyRun_SimpleFileExFlags + 421
      17 org.python.python 0x001b5a57 Py_Main + 3095
      18 org.python.pythonapp 0x00001fca 0x1000 + 4042

      Thread 1:
      0 libSystem.B.dylib 0x9265fe0e poll$UNIX2003 + 10
      1 libSystem.B.dylib 0x9262a155 _pthread_start + 321
      2 libSystem.B.dylib 0x9262a012 thread_start + 34

      Thread 2:
      0 libSystem.B.dylib 0x9260046e __semwait_signal + 10
      1 libSystem.B.dylib 0x9262adcd pthread_cond_wait$UNIX2003 + 73
      2 libzookeeper_mt.2.dylib 0x00247e9f do_completion + 223
      3 libSystem.B.dylib 0x9262a155 _pthread_start + 321
      4 libSystem.B.dylib 0x9262a012 thread_start + 34

      Thread 0 crashed with X86 Thread State (32-bit):
      eax: 0x00000000 ebx: 0x00239a3c ecx: 0x00000000 edx: 0x00000000
      edi: 0x001efaa0 esi: 0x00000000 ebp: 0xbfffe508 esp: 0xbfffe4f0
      ss: 0x0000001f efl: 0x00010286 eip: 0x002332bd cs: 0x00000017
      ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037
      cr2: 0x00000004

      Binary Images:
      0x1000 - 0x1ffe org.python.pythonapp 2.5.0 (2.5.0a0) <5aa9f0cc36fda395f965e08c96613cf5> /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
      0x119000 - 0x1e5feb org.python.python 2.5 (2.5) <523ba54c654eeed6bc670db2f58a73ab> /System/Library/Frameworks/Python.framework/Versions/2.5/Python
      0x232000 - 0x23ffff +zookeeper.so ??? (???) <77134e53d6dbc7bbcf783b0fc2b16d6e> /Library/Python/2.5/site-packages/zookeeper.so
      0x246000 - 0x255fef +libzookeeper_mt.2.dylib ??? (???) <839ce6d5a904236d0f0112e75656ecfd> /usr/local/lib/libzookeeper_mt.2.dylib
      0x8fe00000 - 0x8fe2db43 dyld 97.1 (???) <458eed38a009e5658a79579e7bc26603> /usr/lib/dyld
      0x925f8000 - 0x9275fff3 libSystem.B.dylib ??? (???) <ae47ca9b1686b065f8ac4d2de09cc432> /usr/lib/libSystem.B.dylib
      0x96aea000 - 0x96af1fe9 libgcc_s.1.dylib ??? (???) <e280ddf3f5fb3049e674edcb109f389a> /usr/lib/libgcc_s.1.dylib
      0x96fc9000 - 0x96fcdfff libmathCommon.A.dylib ??? (???) /usr/lib/system/libmathCommon.A.dylib
      0xffff0000 - 0xffff1780 libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

      Attachments

        1. connection_test.py
          0.4 kB
          Josh Fraser

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jfray Josh Fraser
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: