[FLINK-34430] Akka frame size exceeded with many ByteStreamStateHandle being used

• Type: Bug
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.3, 1.17.2, 1.19.0, 1.18.1
• Component/s: Runtime / Coordination
• Assignee: Chesnay Schepler
• Reporter: Piotr Nowojski
• Created: 12/Feb/24 13:06
• Updated: 20/Feb/24 14:48

Description

The following error can happen

Discarding oversized payload sent to Actor[akka.tcp://flink@xxxx/user/rpc/taskmanager_0#-yyyy]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 11212046 bytes.
	
error.stack_trace
akka.remote.OversizedPayloadException: Discarding oversized payload sent to Actor[akka.tcp://flink@xxxx/user/rpc/taskmanager_0#-yyyy]: max allowed size 10485760 bytes, actual size of encoded class org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation was 11212046 bytes.

when https://issues.apache.org/jira/browse/FLINK-26050 is causing large amount of small sst files to be created and never deleted. If those files are small enough to be handled by ByteStreamStateHandle akka frame size can be exceeded.

Issue Links

• is related to: FLINK-26050 Too many small sst files in rocksdb state backend when using time window created in ascending order
• is related to: FLINK-4399 Add support for oversized messages