Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-430

RaftLogCache#getCachedSegmentNum hits ConcurrentModificationException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 0.3.0
    • server
    • None

    Description

      While running performance runs with Ozone, it hits ConcurrentModificationException while loading the cached Raft Segment

      2018-11-16 14:55:56,329 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
      
      2018-11-16 14:55:56,588 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
      
      2018-11-16 14:55:56,778 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3962
      
      2018-11-16 14:55:56,870 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3963
      
      2018-11-16 14:55:56,895 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
      
      2018-11-16 14:55:56,896 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3964
      
      2018-11-16 14:55:56,898 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3965
      
      2018-11-16 14:55:56,899 ERROR org.apache.ratis.server.impl.LogAppender: GrpcLogAppender(e3e9a703-55bb-482b-a0a1-ce8000474ac2 -> 0813f1a9-61be-4cab-aa05-d5640f4a8341) unexpected exception
      
      java.util.ConcurrentModificationException
      
              at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)
      
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
      
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      
              at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
      
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      
              at java.util.stream.LongPipeline.reduce(LongPipeline.java:438)
      
              at java.util.stream.LongPipeline.sum(LongPipeline.java:396)
      
              at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526)
      
              at org.apache.ratis.server.storage.RaftLogCache.getCachedSegmentNum(RaftLogCache.java:118)
      
              at org.apache.ratis.server.storage.RaftLogCache.shouldEvict(RaftLogCache.java:122)
      
              at org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:215)
      
              at org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:193)
      
              at org.apache.ratis.server.storage.SegmentedRaftLog.getEntryWithData(SegmentedRaftLog.java:199)
      
              at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:207)
      
              at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
      
              at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
      
              at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:100)
      
              at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        1. r430_20181120.patch
          21 kB
          Tsz-wo Sze
        2. r430_20181121.patch
          24 kB
          Tsz-wo Sze

        Activity

          People

            szetszwo Tsz-wo Sze
            shashikant Shashikant Banerjee
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: