[ZOOKEEPER-1936] Server exits when unable to create data directory due to race

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.6, 3.5.0
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Ted Yu
• Reporter: Harald Musum
• Created: 10/Jun/14 07:25
• Updated: 24/Jan/20 14:20
• Resolved: 23/Jan/20 15:47

Description

We sometime see issues with ZooKeeper server not starting and seeing this error in the log:

[2014-05-27 09:29:48.248] ERROR : -
.org.apache.zookeeper.server.ZooKeeperServerMain Unexpected exception,
exiting abnormally\nexception=\njava.io.IOException: Unable to create data
directory /home/y/var/zookeeper/version-2\n\tat
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:85)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)\n\tat
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)\n\tat
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)\n\tat
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)\n\t
[...]

Stack trace from JVM gives this:

"PurgeTask" daemon prio=10 tid=0x000000000201d000 nid=0x1727 runnable
[0x00007f55d7dc7000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createDirectory(Native Method)
at java.io.File.mkdir(File.java:1310)
at java.io.File.mkdirs(File.java:1337)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:84)
at org.apache.zookeeper.server.PurgeTxnLog.purge(PurgeTxnLog.java:68)
at
org.apache.zookeeper.server.DatadirCleanupManager$PurgeTask.run(DatadirCleanupManager.java:140)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

"zookeeper server" prio=10 tid=0x00000000027df800 nid=0x1715 runnable
[0x00007f55d7ed8000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.createDirectory(Native Method)
at java.io.File.mkdir(File.java:1310)
at java.io.File.mkdirs(File.java:1337)
at
org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:84)
at
org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:103)
at
org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
at
org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
[...]

So it seems that when autopurge is used (as it is in our case), it might happen at the same time as starting the server itself. In FileTxnSnapLog() it will check if the directory exists and create it if not. These two tasks do this at the same time, and mkdir fails and server exits the JVM.

Issue Links

• links to: GitHub Pull Request #75
• links to: GitHub Pull Request #1225