Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-542

When a server restarts, purge(-1) causes IllegalStateException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.4.0
    • server
    • None

    Description

      IllegalStateException stating "Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0]" when ozone datanode is restarted.

      2019-05-05 06:26:47,217 INFO org.apache.ratis.server.impl.RaftServerProxy: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: remove LEADER group-AC345A757F02 bf6fd9c4-100a-4035-98db-abe8e0a25fb9:t2, leader=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, voted=bf6fd9c4-100a-4035-98db-abe8e0a25fb9, raftlog=bf6fd9c4-100a-4035-98db-abe8e0a25fb9-SegmentedRaftLog:OPENED:c2,f2,i2, conf=1: [bf6fd9c4-100a-4035-98db-abe8e0a25fb9:172.27.76.77:9858], old=null RUNNING
      2019-05-05 06:26:47,230 INFO org.apache.ratis.server.impl.RaftServerImpl: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown group-AC345A757F02
      2019-05-05 06:26:47,230 INFO org.apache.ratis.util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-AC345A757F02,id=bf6fd9c4-100a-4035-98db-abe8e0a25fb9
      2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.RoleInfo: bf6fd9c4-100a-4035-98db-abe8e0a25fb9: shutdown LeaderState
      2019-05-05 06:26:47,231 INFO org.apache.ratis.server.impl.PendingRequests: bf6fd9c4-100a-4035-98db-abe8e0a25fb9-PendingRequests: sendNotLeaderResponses
      2019-05-05 06:26:47,236 INFO org.apache.ratis.server.impl.StateMachineUpdater: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: set stopIndex = 2
      2019-05-05 06:26:47,239 ERROR org.apache.ratis.server.impl.StateMachineUpdater: Terminating with exit status 2: StateMachineUpdater-bf6fd9c4-100a-4035-98db-abe8e0a25fb9-group-AC345A757F02: the StateMachineUpdater hits Throwable
      java.lang.IllegalStateException: Unexpected gap in segments: binarySearch(-1) returns -1, segments=[log-0_0]
      at org.apache.ratis.server.storage.RaftLogCache$LogSegmentList.purge(RaftLogCache.java:259)
      at org.apache.ratis.server.storage.RaftLogCache.purge(RaftLogCache.java:461)
      at org.apache.ratis.server.storage.SegmentedRaftLog.purgeImpl(SegmentedRaftLog.java:273)
      at org.apache.ratis.server.storage.RaftLog.purge(RaftLog.java:325)
      at org.apache.ratis.server.impl.StateMachineUpdater.run(StateMachineUpdater.java:187)
      at java.lang.Thread.run(Thread.java:748)
      2019-05-05 06:26:47,249 WARN org.apache.hadoop.fs.CachingGetSpaceUsed: Thread Interrupted waiting to refresh disk information: sleep interrupted
      

      Attachments

        1. r542_20190509.patch
          13 kB
          Tsz-wo Sze
        2. r542_20190510.patch
          13 kB
          Tsz-wo Sze
        3. r542_20190513.patch
          14 kB
          Tsz-wo Sze
        4. r542_20190514.patch
          13 kB
          Tsz-wo Sze
        5. r542_20190515.patch
          16 kB
          Tsz-wo Sze
        6. r542_20190515b.patch
          16 kB
          Tsz-wo Sze
        7. r542_20190516.patch
          16 kB
          Tsz-wo Sze

        Activity

          People

            szetszwo Tsz-wo Sze
            ljain Lokesh Jain
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: