[ZOOKEEPER-939] the threads number of a zookeeper is increased all the time

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.3.0
• Component/s: server

Environment

Linux 2.6.9-52bs #2 SMP Fri Jan 26 13:34:38 CST 2007 x86_64 x86_64 x86_64 GNU/Linux

• Tags: server
• Assignee: Unassigned
• Reporter: Qian Ye
• Created: 24/Nov/10 07:53
• Updated: 06/Sep/11 03:11
• Resolved: 06/Sep/11 03:10

Description

I have a group of zookeeper servers, there are three servers in this group.
server.0=10.81.4.11:2888:3888
server.1=10.23.240.93:2888:3888
server.2=10.23.244.224:2888:3888

At first, the cluster ran well. About several days ago, I shut down the zookeeper process on one of servers(server.2)., and today, I find that the other two servers run in wired status(the network is fine). The zookeeper process take pretty much resource on the two servers:

on server.1 (it's the leader)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26836 work 18 0 12.8g 803m 8724 S 3.7 10.1 195:56.56 java

$ ll /proc/26836/fd/ | wc -l
3586

[[e-mail] conf]$ ll /proc/26836/task/ | wc -l
10510

some warning log:
2010-11-24 15:37:48,705 - WARN [Thread-37409:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:39:48,626 - WARN [Thread-37414:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:39:48,656 - WARN [Thread-37413:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:39:48,657 - WARN [Thread-37413:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:41:48,614 - WARN [Thread-37417:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:41:48,643 - WARN [Thread-37418:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:41:48,662 - WARN [Thread-37417:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:43:48,627 - WARN [Thread-37421:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:43:48,627 - WARN [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:43:48,654 - WARN [Thread-37421:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:44:48,622 - WARN [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:44:48,652 - WARN [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:44:48,653 - WARN [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:45:48,668 - WARN [Thread-37426:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:46:48,647 - WARN [Thread-37427:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:46:48,722 - WARN [Thread-37428:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:46:48,736 - WARN [Thread-37427:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:47:48,687 - WARN [Thread-37430:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)

on server.0
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27322 work 19 0 15.2g 943m 9140 S 38.6 11.8 1396:51 java

$ ll /proc/27322/fd/ | wc -l
3587

$ ll /proc/27322/task/ | wc -l
12938

2010-11-24 15:37:49,269 - WARN [Thread-37407:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:39:49,235 - WARN [Thread-37412:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:39:49,410 - WARN [Thread-37411:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:39:49,411 - WARN [Thread-37411:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:41:49,314 - WARN [Thread-37416:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:41:49,383 - WARN [Thread-37415:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:41:49,405 - WARN [Thread-37415:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:43:49,372 - WARN [Thread-37420:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:43:49,512 - WARN [Thread-37419:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:43:49,513 - WARN [Thread-37419:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:44:49,407 - WARN [Thread-37422:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:45:49,645 - WARN [Thread-37424:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:45:49,781 - WARN [Thread-37423:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:45:49,799 - WARN [Thread-37423:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:46:49,495 - WARN [Thread-37427:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-11-24 15:47:49,541 - WARN [Thread-37429:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:263)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:629)
2010-11-24 15:47:49,622 - WARN [Thread-37428:QuorumCnxManager$SendWorker@581] - Interrupted while waiting for message on queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1899)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1976)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:570)
2010-11-24 15:47:49,622 - WARN [Thread-37428:QuorumCnxManager$SendWorker@589] - Send worker leaving thread
2010-11-24 15:48:48,827 - WARN [Thread-37431:QuorumCnxManager$RecvWorker@658] - Connection broken:
java.io.IOException: Channel eof
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)

What's more, the number of threads under the zookeeper process is still increasing time by time. It seems that , something is wrong in communication of the two servers. Have anyone met such problem before?

Issue Links

• is related to: ZOOKEEPER-880 QuorumCnxManager$SendWorker grows without bounds