[RATIS-430] RaftLogCache#getCachedSegmentNum hits ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3.0
• Component/s: server
• Target Version/s: 0.4.0
• Assignee: Tsz-wo Sze
• Reporter: Shashikant Banerjee
• Created: 19/Nov/18 09:35
• Updated: 22/Mar/19 05:27
• Resolved: 22/Nov/18 19:44

Description

While running performance runs with Ozone, it hits ConcurrentModificationException while loading the cached Raft Segment

2018-11-16 14:55:56,329 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

2018-11-16 14:55:56,588 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

2018-11-16 14:55:56,778 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3962

2018-11-16 14:55:56,870 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3963

2018-11-16 14:55:56,895 WARN org.apache.ratis.grpc.server.GrpcLogAppender: e3e9a703-55bb-482b-a0a1-ce8000474ac2: Failed appendEntries to 0813f1a9-61be-4cab-aa05-d5640f4a8341:172.26.32.228:9858: org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception

2018-11-16 14:55:56,896 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3964

2018-11-16 14:55:56,898 INFO org.apache.ratis.server.storage.RaftLogWorker: Rolling segment:e3e9a703-55bb-482b-a0a1-ce8000474ac2-RaftLogWorker index to:3965

2018-11-16 14:55:56,899 ERROR org.apache.ratis.server.impl.LogAppender: GrpcLogAppender(e3e9a703-55bb-482b-a0a1-ce8000474ac2 -> 0813f1a9-61be-4cab-aa05-d5640f4a8341) unexpected exception

java.util.ConcurrentModificationException

        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380)

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)

        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)

        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)

        at java.util.stream.LongPipeline.reduce(LongPipeline.java:438)

        at java.util.stream.LongPipeline.sum(LongPipeline.java:396)

        at java.util.stream.ReferencePipeline.count(ReferencePipeline.java:526)

        at org.apache.ratis.server.storage.RaftLogCache.getCachedSegmentNum(RaftLogCache.java:118)

        at org.apache.ratis.server.storage.RaftLogCache.shouldEvict(RaftLogCache.java:122)

        at org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:215)

        at org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:193)

        at org.apache.ratis.server.storage.SegmentedRaftLog.getEntryWithData(SegmentedRaftLog.java:199)

        at org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:207)

        at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)

        at org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)

        at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:100)

        at java.lang.Thread.run(Thread.java:745)