[HDDS-3032] Cannot write 32MB chunks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Ozone Datanode
• Labels: TriagePending
• Assignee: Unassigned
• Reporter: Attila Doroszlai
• Created: 19/Feb/20 07:25
• Updated: 31/Oct/24 17:20
• Resolved: 31/Oct/24 17:20

Description

Writing 32MB chunks fails with various errors.

steps to reproduce

ozone freon dcg -t 1 -n 1 -s 33554432

1. With Ratis 0.5.0-90cd474-SNAPSHOT (used by current Ozone master):

org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 33554432: 33554686
  at org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:524)
  at org.apache.ratis.thirdparty.io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:387)

Which is strange, because Datanode attempts to set max. message size to 32MB + 16KB.

2. With Ratis built locally from current Ratis master (46f255cb):

Caused by: org.apache.ratis.protocol.StateMachineException: org.apache.ratis.server.raftlog.RaftLogIOException from Server ccb25fbf-9bd1-4094-a632-00f4168213bb@group-B1FA90A78F31: Log entry size 33554666 exceeds the max buffer limit of 33554432
	at org.apache.ratis.server.raftlog.RaftLog.appendImpl(RaftLog.java:178)
	at org.apache.ratis.server.raftlog.RaftLog.lambda$append$2(RaftLog.java:157)
	at org.apache.ratis.server.raftlog.RaftLogSequentialOps$Runner.runSequentially(RaftLogSequentialOps.java:68)
	at org.apache.ratis.server.raftlog.RaftLog.append(RaftLog.java:157)
	at org.apache.ratis.server.impl.ServerState.appendLog(ServerState.java:282)
	at org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:518)
	at org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:604)

With ozone.scm.chunk.size=32MB setting, ozone freon ockg -n 1 -t 1 -s 33554432 also fails, but without apparent errors in the datanode log.

Issue Links

• is related to: HDDS-11295 Grpc message should be larger than appender buffer size