Description
While running performance runs with Ozone, it hits ConcurrentModificationException while loading the cached Raft Segment
2018-11-16 14:55:56,329 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-11-16 14:55:56,588 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-11-16 14:55:56,778 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3962 2018-11-16 14:55:56,870 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3963 2018-11-16 14:55:56,895 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception 2018-11-16 14:55:56,896 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3964 2018-11-16 14:55:56,898 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3965 2018-11-16 14:55:56,899 ERROR org.apache.ratis.server.impl.LogAppender: GrpcLogAppender(e3e9a703-55bb-482b-a0a1-ce8000474ac2 -> 0813f1a9-61be-4cab-aa05-d5640f4a8341) unexpected exception java.util.ConcurrentModificationException at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.LongPipeline.reduce(LongPipeline.java:438) at java.util.stream.LongPipeline.sum(LongPipeline.java:396) at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526) at org.apache.ratis.server.storage.RaftLogCache.getCachedSegmentNum(RaftLogCache.java:118) at org.apache.ratis.server.storage.RaftLogCache.shouldEvict(RaftLogCache.java:122) at org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:215) at org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:193) at org.apache.ratis.server.storage.SegmentedRaftLog.getEntryWithData(SegmentedRaftLog.java:199) at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:207) at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152) at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96) at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:100) at java.lang.Thread.run(Thread.java:745)