[ZOOKEEPER-3082] Fix server snapshot behavior when out of disk space

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.6.0, 3.4.12, 3.5.5
• Fix Version/s: 3.6.0
• Component/s: server
• Labels: pull-request-available
• Assignee: Brian Nixon
• Reporter: Brian Nixon
• Created: 05/Jul/18 18:53
• Updated: 04/Oct/19 14:55
• Resolved: 30/Jul/18 04:23

Description

When the ZK server tries to make a snapshot and the machine is out of disk space, the snapshot creation fails and throws an IOException. An empty snapshot file is created, (probably because the server is able to create an entry in the dir) but is not able to write to the file.
 
If snapshot creation fails, the server commits suicide. When it restarts, it will do so from the last known good snapshot. However, when it tries to make a snapshot again, the same thing happens. This results in lots of empty snapshot files being created. If eventually the DataDirCleanupManager garbage collects the good snapshot files then only the empty files remain. At this point, the server is well and truly screwed.

Issue Links

• links to: GitHub Pull Request #560