[ZOOKEEPER-2925] ZooKeeper server fails to start on first-startup due to race to create dataDir & snapDir

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.6
• Fix Version/s: 3.4.10
• Component/s: other
• Labels: easyfix newbie patch
• Assignee: Unassigned
• Reporter: Robert P. Thille
• Created: 27/Oct/17 22:16
• Updated: 04/Oct/19 14:55

Description

Due to two threads trying to create the dataDir and snapDir, and the java.io.File.mkdirs() call returning false both for errors and for the directory already existing, sometimes ZooKeeper will fail to start with the following stack trace:

2017-10-25 22:30:40,069 [myid:] - INFO  [main:ZooKeeperServerMain@95] - Starting server
2017-10-25 22:30:40,075 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.6-mdavis8efb625--1, built on 10/25/2017 01:12 GMT

[ More 'Server environment:blah blah blah' messages trimmed]

2017-10-25 22:30:40,077 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=/
2017-10-25 22:30:40,081 [myid:] - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
java.io.IOException: Unable to create data directory /bp2/data/version-2
    at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:85)
    at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:104)
    at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
    at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
    at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2017-10-25 22:30:40,085 [myid:] - INFO  [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed.

this is caused by the QuorumPeerMain thread and the PurgeTask thread both competing to create the directories.