[BEAM-10458] Message limit for grpc message is set to default 4 MB

• Type: Bug
• Status: Triage Needed
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.22.0
• Component/s: sdk-py-harness
• Assignee: Unassigned
• Reporter: Keshav
• Created: 10/Jul/20 20:50
• Updated: 01/Sep/24 06:55

Description

The default limit of a grpc message is set low to 4 MB, and is not overridden in the sdk_worker.py for the _control_channel. This was first discovered when submitting a beam job via a flink operator orchestrated by kubernetes (https://github.com/GoogleCloudPlatform/flink-on-k8s-operator). This issue was resolved for apache-beam==2.18 in BEAM-10334, but exists in apache-beam==2.22 (and presumably all versions 2.17 onwards). 

 

This can be fixed by modifying the options parameter to allow any size grpc message to be sent or received.

Issue Links

• links to: GitHub Pull Request #12228