Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.18.0, 2.22.0
-
None
Description
The default limit of a grpc message is set low to 4 MB, and is not overridden in the sdk_worker.py for the _control_channel. This was first discovered when submitting a beam job via a flink operator orchestrated by kubernetes (https://github.com/GoogleCloudPlatform/flink-on-k8s-operator). The below error (for apache-beam 2.18.0) was seen.
2020/06/24 18:37:28 Executing: python -m apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 221, in <module> main(sys.argv) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 117, in run for work_request in control_stub.Control(get_responses()): File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (4847139 vs. 4194304)" debug_error_string = "{"created":"@1593023849.325990420","description":"Received message larger than max (4847139 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
This can be fixed by modifying the options parameter to allow any size grpc message to be sent or received.
Attachments
Issue Links
- links to
