Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10334

Message limit for grpc message is set to default 4 MB

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.18.0, 2.22.0
    • 2.23.0
    • sdk-py-harness
    • None

    Description

      The default limit of a grpc message is set low to 4 MB, and is not overridden in the sdk_worker.py for the _control_channel. This was first discovered when submitting a beam job via a flink operator orchestrated by kubernetes (https://github.com/GoogleCloudPlatform/flink-on-k8s-operator). The below error (for apache-beam 2.18.0) was seen. 

       

      2020/06/24 18:37:28 Executing: python -m apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 221, in <module> main(sys.argv) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 117, in run for work_request in control_stub.Control(get_responses()): File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (4847139 vs. 4194304)" debug_error_string = "{"created":"@1593023849.325990420","description":"Received message larger than max (4847139 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
      

       

      This can be fixed by modifying the options parameter to allow any size grpc message to be sent or received.

      Attachments

        Issue Links

          Activity

            People

              kshivvy Keshav
              kshivvy Keshav
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h