[ZOOKEEPER-2714] Zookeeper (standalone) failed to start up

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Cannot Reproduce
• Affects Version/s: 3.4.6
• Component/s: server

Environment

LINUX

• Assignee: Unassigned
• Reporter: Daniel C
• Created: 08/Mar/17 23:17
• Updated: 09/Mar/17 16:48
• Resolved: 09/Mar/17 16:48

Description

We've a standalone ZK setup. Upon restart, it failed to serve requests.

Here are the logs:
------------------
2017-03-05 17:33:58,888 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /zookeeper/zookeeper-3.4.6/conf/zoo.1.cfg
2017-03-05 17:33:58,898 [myid:] - WARN [main:QuorumPeerConfig@293] - No server failure will be tolerated. You need at least 3 servers.
2017-03-05 17:33:58,898 [myid:] - INFO [main:QuorumPeerConfig@340] - Defaulting to majority quorums
2017-03-05 17:33:58,909 [myid:1] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 10
2017-03-05 17:33:58,910 [myid:1] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 5
2017-03-05 17:33:58,911 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started.
2017-03-05 17:33:58,946 [myid:1] - INFO [main:QuorumPeerMain@127] - Starting quorum peer
2017-03-05 17:33:58,966 [myid:1] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.
2017-03-05 17:33:58,991 [myid:1] - INFO [main:NIOServerCnxnFactory@94] - binding to port 0.0.0.0/0.0.0.0:2181
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@959] - tickTime set to 2000
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@979] - minSessionTimeout set to -1
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@990] - maxSessionTimeout set to -1
2017-03-05 17:33:59,016 [myid:1] - INFO [main:QuorumPeer@1005] - initLimit set to 20
2017-03-05 17:34:01,328 [myid:1] - INFO [main:QuorumPeer@473] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-03-05 17:34:01,332 [myid:1] - INFO [main:QuorumPeer@488] - acceptedEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation
2017-03-05 17:34:01,335 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48198
2017-03-05 17:34:01,339 [myid:1] - INFO [Thread-4:QuorumCnxManager$Listener@504] - My election bind port: server001-internal/10.245.66.137:3888
2017-03-05 17:34:01,346 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,346 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48198 (no session established for client)
2017-03-05 17:34:01,346 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48199
2017-03-05 17:34:01,347 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,347 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48199 (no session established for client)
2017-03-05 17:34:01,347 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48200
2017-03-05 17:34:01,347 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.245.66.147:48200 (no session established for client)
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.147:48201
2017-03-05 17:34:01,348 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.245.66.137:46628
2017-03-05 17:34:01,348 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
------------------

Is it a race issue during startup?
2017-03-05 17:34:01,346 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running