Major Description
IllegalStateException stating "Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0]" when ozone datanode is restarted.
2019-05-05 06:26:47,217 INFO org.apache.ratis.server.impl.RaftServerProxy: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: remove LEADER group-AC345A757F02 bf6fd9c4-100a-4035-98db-abe8e0a25fb9:t2, leader=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, voted=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, raftlog=bf6fd9c4-100a-4035-98db-abe8e0a25fb9-SegmentedRaftLog:OPENED:c2,f2,i2, conf=1: [bf6fd9c4-100a-4035-98db-abe8e0a25fb9:172.27.76.77:9858], old=null RUNNING 2019-05-05 06:26:47,230 INFO org.apache.ratis.server.impl.RaftServerImpl: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown group-AC345A757F02 2019-05-05 06:26:47,230 INFO org.apache.ratis.util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-AC345A757F02,id=bf6fd9c4-100a-4035-98db-abe8e0a25fb9 2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.RoleInfo: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown LeaderState 2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.PendingRequests: bf6fd9c4-100a-4035-98db-abe8e0a25fb9-PendingRequests: sendNotLeaderResponses 2019-05-05 06:26:47,236 INFO org.apache.ratis.server.impl.StateMachineUpdater: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: set stopIndex = 2 2019-05-05 06:26:47,239 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating with exit status 2: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: the StateMachineUpdater hits Throwable java.lang.IllegalStateException: Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0] at org.apache.ratis.server.storage.RaftLogCache$LogSegmentList.purge(RaftLogCache.java:259) at org.apache.ratis.server.storage.RaftLogCache.purge(RaftLogCache.java:461) at org.apache.ratis.server.storage.SegmentedRaftLog.purgeImpl(SegmentedRaftLog.java:273) at org.apache.ratis.server.storage.RaftLog.purge(RaftLog.java:325) at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:187) at java.lang.Thread.run(Thread.java:748) 2019-05-05 06:26:47,249 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted