[ZOOKEEPER-893] ZooKeeper high cpu usage when invalid requests

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.3.2, 3.4.0
• Component/s: server

Environment

Linux 2.6.16
4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode)

• Hadoop Flags: Reviewed
• Tags: zookeeper server cpu ZOOKEEPER-427
• Assignee: Thijs Terlouw
• Reporter: Thijs Terlouw
• Created: 11/Oct/10 07:15
• Updated: 23/Nov/11 19:22
• Resolved: 19/Oct/10 22:38

Description

When ZooKeeper receives certain illegally formed messages on the internal communication port (:4181 by default), it's possible for ZooKeeper to enter an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, but that patch does not resolve all issues.

from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java

the two affected parts:
===========
int length = msgLength.getInt();
if(length <= 0) {
throw new IOException("Invalid packet length:" + length);
}
===========

===========
while (message.hasRemaining()) {
temp_numbytes = channel.read(message);
if(temp_numbytes < 0)

{ throw new IOException("Channel eof before end"); }

numbytes += temp_numbytes;
}
===========

how to replicate this bug:

perform an nmap portscan against your zookeeper server: "nmap -sV -n your.ip.here -p4181"
wait for a while untill you see some messages in the logfile and then you will see 100% cpu usage. It does not recover from this situation. With my patch, it does not occur anymore

Issue Links

• is related to: ZOOKEEPER-427 ZooKeeper server unexpectedly high CPU utilisation