Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
3.5.3
-
None
-
I deploy a zookeeper cluster on three nodes. And enable ssl capability under below guidline:https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide
And i use zookeeper client which also enable ssl capability to connect this zookeeper server and set same data to two node under below demo:
CuratorFramework client;
// each time we instance a new zookeeper client ....
String path1, path2;
// instance path1, path2 ......
String status = "ok"
client.client.setData().forPath(path1,status.getBytes());
client.client.setData().forPath(path2,status.getBytes());
// close zookeeper client.......
This function will be called each five seconds and it work good while ssl capability disabled. when ssl capability enabled, zookeeper server run about one day, and an outofmemory error will occurred and auto produce java_pidXXX.hprof file by zookeeper process . i use Eclipse Memory Analyzer to analize the hprof file and found instance of DataTree used more than six handreds MB memory and more than eighty seven percent memory used by dataTree's field which name is dataWatches. And i use four letter command to check and found too many watches on all of this three nodes. I guess those too many watches cause the error but i don't know why there are so many watches!
Additional, if disabled the ssl capability. use four letter command and can only found there are several watches on each node. and count of watches will not increased.
Each zookeeper node run under VM which has eight core and eight GB memory, and it's os are centos6.5/centos7.3/redhat6.5/redhat7 and run zookeeper and this demo with JDK1.8.
This issue will happened under zookeeper 3.5.1 and 3.5.2 and 3.5.3.
.......
I deploy a zookeeper cluster on three nodes. And enable ssl capability under below guidline: https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide And i use zookeeper client which also enable ssl capability to connect this zookeeper server and set same data to two node under below demo: CuratorFramework client; // each time we instance a new zookeeper client .... String path1, path2; // instance path1, path2 ...... String status = "ok" client.client.setData().forPath(path1,status.getBytes()); client.client.setData().forPath(path2,status.getBytes()); // close zookeeper client....... This function will be called each five seconds and it work good while ssl capability disabled. when ssl capability enabled, zookeeper server run about one day, and an outofmemory error will occurred and auto produce java_pidXXX.hprof file by zookeeper process . i use Eclipse Memory Analyzer to analize the hprof file and found instance of DataTree used more than six handreds MB memory and more than eighty seven percent memory used by dataTree's field which name is dataWatches. And i use four letter command to check and found too many watches on all of this three nodes. I guess those too many watches cause the error but i don't know why there are so many watches! Additional, if disabled the ssl capability. use four letter command and can only found there are several watches on each node. and count of watches will not increased. Each zookeeper node run under VM which has eight core and eight GB memory, and it's os are centos6.5/centos7.3/redhat6.5/redhat7 and run zookeeper and this demo with JDK1.8. This issue will happened under zookeeper 3.5.1 and 3.5.2 and 3.5.3. .......
Description
When use SSL on zookeeper server, counts of watches may increase more than forty thousands and lead zoookeeper process outofmemroy error after zookeeper server started one day.
check command:
echo wchs | nc localhost 2181
check result:
[zookeeper@localhost bin]$ echo wchs | nc localhost 2181
44412 connections watching 1 paths
Total watches:44412