[HDDS-8977] Ratis crash if a lot of directories deleted at once

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Sumit Agrawal
• Reporter: Ivan Zlenko
• Created: 04/Jul/23 11:27
• Updated: 2 days ago 20:12
• Resolved: 29/Aug/23 07:23

Description

DirectoryDeletingService periodically scans for deleted directories and sends a message containing all deleted directories.
There is no checks for this message before it will be sent. So if we delete enough directory we can force Ratis to crash because message with list of directories would be to large to be handled by Ratis:

2023-06-30 16:49:25,822 [DirectoryDeletingService#0] ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer: Unknown error occurs org.apache.ratis.server.raftlog.RaftLogIOException: Log entry size 41464041 exceeds the max buffer limit of 33554432 at org.apache.ratis.server.raftlog.RaftLogBase.appendImpl(RaftLogBase.java:184) at org.apache.ratis.server.raftlog.RaftLogBase.lambda$append$2(RaftLogBase.java:161) at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:69) at org.apache.ratis.server.raftlog.RaftLogBase.append(RaftLogBase.java:161) at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:366) at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:770) at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:878) at org.apache.ratis.server.impl.RaftServerImpl.lambda$null$12(RaftServerImpl.java:815) at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:117) at org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitClientRequestAsync$13(RaftServerImpl.java:815) at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750)

 

Issue Links

• is related to: HDDS-11508 Decouple delete batch limits from Ratis request size
• relates to: RATIS-1783 MAX_OP size is not configurable on raft log read
• links to: GitHub Pull Request #5207