[ZOOKEEPER-3781] Zookeeper 3.5.7 not creating snapshot

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.5.10, 3.7.0, 3.8.0, 3.6.4
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Wei Chu
• Created: 01/Apr/20 14:54
• Updated: 16/May/22 19:13
• Resolved: 06/Mar/21 19:43

Description

Hi,

In our deployment, we are using zookeepers as Kafka quorum.

Recently we upgraded Zookeeper from 3.4.14 to 3.5.7.
During the upgrade, we enabled snapshot.trust.empty and removed it after.
However, about 25% of zookeepers (all on 3.5.7) are not creating snapshot files, so they keep crashing on reboot with ERROR.

ERROR [main:QuorumPeer@937] - Unable to load database on disk
<http://java.io|java.io>.IOException: No snapshot found, but there are log entries. Something is broken!

Nothing suspecious did I find from the zookeeper log.

I managed to reproduce the issue with the data from our servers, I put them in the attachment.

Issue Links

• is related to: ZOOKEEPER-4421 Backport ZOOKEEPER-3781 for branch-3.5
• links to: GitHub Pull Request #1581
• links to: GitHub Pull Request #1876