[ZOOKEEPER-2570] ZooKeeper clients are timed out when ZooKeeper servers are very busy

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Assignee: Mohammad Arshad
• Reporter: Mohammad Arshad
• Created: 08/Sep/16 16:27
• Updated: 13/Oct/16 18:59

Description

ZooKeeper clients are timed out when ZooKeeper servers are very busy. Clients throw below exception and fail all the pending operations

org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)

Clients log bellow information

2016-09-22 01:49:08,001 [myid:127.0.0.1:11228] - WARN  [main-SendThread(127.0.0.1:11228):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 13908ms for sessionid 0x20000d21b280000
2016-09-22 01:49:08,001 [myid:127.0.0.1:11228] - INFO  [main-SendThread(127.0.0.1:11228):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 13908ms for sessionid 0x20000d21b280000, closing socket connection and attempting reconnect

STEPS TO REPRODECE:

1. Create multi operation

   List<Op> ops = new ArrayList<Op>();
           for (int i = 0; i < N; i++) {
               Op create = Op.create(rootNode + "/" + i, "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE,
                       CreateMode.PERSISTENT);
               ops.add(create);
           }
   

   Chose N in such a way that the total multi operation request size is less than but near 1 MB. For bigger request size increase jute.maxbuffer in servers

2. Submit the multi operation request

    zooKeeper.multi(ops);

3. After repeating above steps few times issue is reproduced