[RATIS-542] When a server restarts, purge(-1) causes IllegalStateException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: server
• Target Version/s: 0.4.0
• Assignee: Tsz-wo Sze
• Reporter: Lokesh Jain
• Created: 05/May/19 06:33
• Updated: 16/May/19 22:23
• Resolved: 16/May/19 22:23

Description

IllegalStateException stating "Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0]" when ozone datanode is restarted.

2019-05-05 06:26:47,217 INFO org.apache.ratis.server.impl.RaftServerProxy: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: remove LEADER group-AC345A757F02 bf6fd9c4-100a-4035-98db-abe8e0a25fb9:t2, leader=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, voted=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, raftlog=bf6fd9c4-100a-4035-98db-abe8e0a25fb9-SegmentedRaftLog:OPENED:c2,f2,i2, conf=1: [bf6fd9c4-100a-4035-98db-abe8e0a25fb9:172.27.76.77:9858], old=null RUNNING
2019-05-05 06:26:47,230 INFO org.apache.ratis.server.impl.RaftServerImpl: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown group-AC345A757F02
2019-05-05 06:26:47,230 INFO org.apache.ratis.util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-AC345A757F02,id=bf6fd9c4-100a-4035-98db-abe8e0a25fb9
2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.RoleInfo: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown LeaderState
2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.PendingRequests: bf6fd9c4-100a-4035-98db-abe8e0a25fb9-PendingRequests: sendNotLeaderResponses
2019-05-05 06:26:47,236 INFO org.apache.ratis.server.impl.StateMachineUpdater: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: set stopIndex = 2
2019-05-05 06:26:47,239 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating with exit status 2: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: the StateMachineUpdater hits Throwable
java.lang.IllegalStateException: Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0]
at org.apache.ratis.server.storage.RaftLogCache$LogSegmentList.purge(RaftLogCache.java:259)
at org.apache.ratis.server.storage.RaftLogCache.purge(RaftLogCache.java:461)
at org.apache.ratis.server.storage.SegmentedRaftLog.purgeImpl(SegmentedRaftLog.java:273)
at org.apache.ratis.server.storage.RaftLog.purge(RaftLog.java:325)
at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:187)
at java.lang.Thread.run(Thread.java:748)
2019-05-05 06:26:47,249 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted