[ZOOKEEPER-2307] ZooKeeper not starting because acceptedEpoch is less than the currentEpoch

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.6.3, 3.7.0, 3.5.8
• Component/s: server
• Labels: pull-request-available
• Assignee: Mohammad Arshad
• Reporter: Mohammad Arshad
• Created: 29/Oct/15 15:12
• Updated: 31/Oct/21 08:43
• Resolved: 17/Dec/19 12:48

Description

This issue occurred in one of our test environment where disk was being changed to read only very frequently.
The the scenario is as follows:

1. Configure three node ZooKeeper cluster, lets say nodes are A, B and C
2. Start A and B. Both A and B start successfully, quorum is running.
3. Start C, because of IO error C fails to update acceptedEpoch file. But C also starts successfully, joins the quorum as follower
4. Stop C
5. Start C, bellow exception with message "The accepted epoch, 0 is less than the current epoch, 1" is thrown

   2015-10-29 16:52:32,942 [myid:3] - ERROR [main:QuorumPeer@784] - Unable to load database on disk
   java.io.IOException: The accepted epoch, 0 is less than the current epoch, 1
   	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:781)
   	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:720)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:202)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:139)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:88)
   2015-10-29 16:52:32,946 [myid:3] - ERROR [main:QuorumPeerMain@111] - Unexpected exception, exiting abnormally
   java.lang.RuntimeException: Unable to run quorum server 
   	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:785)
   	at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:720)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:202)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:139)
   	at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:88)
   Caused by: java.io.IOException: The accepted epoch, 0 is less than the current epoch, 1
   	at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:781)
   

Issue Links

• duplicates: ZOOKEEPER-2660 acceptedEpoch and currentEpoch data inconsistency, ZK process can not start!
• is related to: ZOOKEEPER-4222 Backport ZOOKEEPER-2307 to branch-3.6
• is related to: ZOOKEEPER-2162 infinite exception loop occurs when dataDir is lost
• links to: GitHub Pull Request #1145
• links to: GitHub Pull Request #1613