[ZOOKEEPER-3513] Zookeeper upgrade fails due to missing snapshots

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.5.4, 3.6.0
• Component/s: server
• Assignee: Unassigned
• Reporter: Stephan Huttenhuis
• Created: 21/Aug/19 08:18
• Updated: 04/Dec/19 14:16
• Resolved: 04/Dec/19 14:16

Description

In ZOOKEEPER-2325 a check was added that requires a snapshot when loading data. We have been running 3-node ensembles on Zookeeper 3.4.13 for about 5 months for use with Solr Cloud. During this time some ensembles created a few snapshots but other didn't generate any. Because of this upgrading to e.g. 3.5.5 fails.

Either it is perfectly possible for Zookeeper data to have no snapshots or something is going wrong with generating snapshots. The ensembles are straightforward.

• The following stack occurs:

  java.io.IOException: No snapshot found, but there are log entries. Something is broken!
  	at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:211)
  	at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:240)
  	at org.apache.zookeeper.server.ZooKeeperServer.loadData(ZooKeeperServer.java:290)
  	at org.apache.zookeeper.server.ZooKeeperServer.startdata(ZooKeeperServer.java:450)
  	at org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:764)
  	at org.apache.zookeeper.server.ServerCnxnFactory.startup(ServerCnxnFactory.java:98)
  	at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:144)
  	at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
  	at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
  	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
  	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
  

• The zoo.cfg

  # The number of milliseconds of each tick
  tickTime=2000
  # The number of ticks that the initial
  # synchronization phase can take
  initLimit=10
  # The number of ticks that can pass between
  # sending a request and getting an acknowledgement
  syncLimit=5
  # the directory where the snapshot is stored.
  # do not use /tmp for storage, /tmp here is just
  # example sakes.
  dataDir=/data/zookeeper/data
  # the port at which the clients will connect
  clientPort=2181
  
  server.1=myserver1:2888:3888
  server.2=myserver2:2888:3888
  server.3=myserver3:2888:3888
  

   

• The contents of /data/zookeeper/data/version-2

  -rw-r--r-- 1 zookeeper zookeeper    1 Aug  7 21:50 acceptedEpoch
  -rw-r--r-- 1 zookeeper zookeeper    1 Aug  8 20:38 currentEpoch
  -rw-r--r-- 1 zookeeper zookeeper  65M Apr  1 14:44 log.1
  -rw-r--r-- 1 zookeeper zookeeper  65M May 15 23:30 log.100000001
  -rw-r--r-- 1 zookeeper zookeeper  65M Jul  3 23:21 log.100001645
  -rw-r--r-- 1 zookeeper zookeeper  65M Aug  8 20:37 log.300000802
  -rw-r--r-- 1 zookeeper zookeeper  65M Aug 20 13:58 log.70000062a
  -rw-r--r-- 1 zookeeper zookeeper  65M Apr  4 21:22 log.f0
  

Issue Links

• is caused by: ZOOKEEPER-2325 Data inconsistency if all snapshots empty or missing
• requires: ZOOKEEPER-3056 Fails to load database with missing snapshot file but valid transaction log file