[ZOOKEEPER-2200] Perl ZooKeeper locks up during heavy load

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.6
• Component/s: c client

Environment

CentOS 5.8

• Assignee: Unassigned
• Reporter: KARA VAN HORN
• Created: 01/Jun/15 19:02
• Updated: 30/Mar/16 16:42

Description

We are using Perl Net::ZooKeeper (0.38) and Net::ZooKeeper::Lock (0.03) libraries. Deadlock appears to occur at the end during lock cleanup activity. Here is a stack dump (sensitive names changed):

Thread 2 (Thread 0x2ac6fbfa3940 (LWP 13292)):
#0 0x00002ac6f5aed654 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00002ac6f5aeb47b in pthread_cond_signal@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#2 0x00002ac6f835539c in _zk_watcher (handle=<value optimized out>, type=2, state=3, path=<value optimized out>, context=0x33f3ce0) at ZooKeeper.xs:179
#3 0x00002ac6f856d942 in do_foreach_watcher (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:279
#4 deliverWatchers (zh=0x33e4fb0, type=2, state=3, path=0x33f3f50 "/lock/cmts/cisco_device1.net-0001851215", list=0x33ed290)
at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zk_hashtable.c:321
#5 0x00002ac6f8564966 in process_completions (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2114
#6 0x00002ac6f856e101 in do_completion (v=<value optimized out>) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:466
#7 0x00002ac6f5ae683d in start_thread (arg=<value optimized out>) at pthread_create.c:301
#8 0x00002ac6f5dd1fcd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ac6f6056af0 (LWP 12972)):
#0 0x00002ac6f5ae7c65 in pthread_join (threadid=47034119371072, thread_return=0x0) at pthread_join.c:89
#1 0x00002ac6f856e7de in adaptor_finish (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/mt_adaptor.c:293
#2 0x00002ac6f8566cdc in zookeeper_close (zh=0x33e4fb0) at /home/myhome/rpm/BUILD/zookeeper-3.4.6/src/c/src/zookeeper.c:2536
#3 0x00002ac6f8357222 in XS_Net__ZooKeeper_DESTROY (my_perl=0x20df010, cv=<value optimized out>) at ZooKeeper.xs:885
#4 0x00002ac6f4b38af6 in Perl_pp_entersub () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#5 0x00002ac6f4adb8d7 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#6 0x00002ac6f4adf720 in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#7 0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#8 0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#9 0x00002ac6f4b6025c in Perl_free_tmps () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#10 0x00002ac6f4adf78a in Perl_call_sv () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#11 0x00002ac6f4b3d3c6 in Perl_sv_clear () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#12 0x00002ac6f4b3db70 in Perl_sv_free () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#13 0x00002ac6f4b3b0e5 in ?? () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#14 0x00002ac6f4b3b141 in Perl_sv_clean_objs () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#15 0x00002ac6f4ae185e in perl_destruct () from /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
#16 0x0000000000401773 in main ()

There are about 4 out of 10,000 processes that end up in deadlock, and according to our web searches, the only reason pthread_cond_signal would lock is due to it waiting on an already destroyed condition.