[RATIS-408] RaftLogWorker$WriteLog#execute fails if a node rejoins the raft ring after missing few transactions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: raft-group
• Assignee: Unassigned
• Reporter: Nandakumar
• Created: 12/Nov/18 14:48
• Updated: 22/Aug/23 05:08
• Resolved: 22/Aug/23 05:08

Description

When we restart one of the node in the raft ring and in meantime if we have processed few transactions, the restarted node fails on RaftLogWorker$WriteLog#execute

2018-11-12 20:09:13,299 ERROR storage.RaftLogWorker (ExitUtils.java:terminate(86)) - Terminating with exit status 1: 866cb1cc-2d3e-44a7-977f-8510870b42ef-RaftLogWorker failed.
java.lang.IllegalStateException: lastWrittenIndex == 0, entry == term: 2
configurationEntry {
  peers {
    id: "2552aca0-7b51-416e-8e52-1370fdcd5fd7"
    address: "xxx.xxx.xxx.xxx:61142"
  }
  peers {
    id: "542ec0a9-2bc5-4ac8-ad82-ba5d0fe2c2ae"
    address: "xxx.xxx.xxx.xxx:61145"
  }
  peers {
    id: "866cb1cc-2d3e-44a7-977f-8510870b42ef"
    address: "xxx.xxx.xxx.xxx:61154"
  }
}

  at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:60)
  at org.apache.ratis.server.storage.RaftLogWorker$WriteLog.execute(RaftLogWorker.java:351)
  at org.apache.ratis.server.storage.RaftLogWorker.run(RaftLogWorker.java:216)
  at java.lang.Thread.run(Thread.java:748)

At this time:
lastWrittenIndex: 0
entry.getIndex(): 0
entry.getTerm(): 2