Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10334

Message limit for grpc message is set to default 4 MB

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.18.0, 2.22.0
    • 2.23.0
    • sdk-py-harness
    • None

    Description

      The default limit of a grpc message is set low to 4 MB, and is not overridden in the sdk_worker.py for the _control_channel. This was first discovered when submitting a beam job via a flink operator orchestrated by kubernetes (https://github.com/GoogleCloudPlatform/flink-on-k8s-operator). The below error (for apache-beam 2.18.0) was seen. 

       

      2020/06/24 18:37:28 Executing: python -m apache_beam.runners.worker.sdk_worker_main Traceback (most recent call last): File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 221, in <module> main(sys.argv) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker_main.py", line 155, in main sdk_pipeline_options.view_as(ProfilingOptions)) File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py", line 117, in run for work_request in control_stub.Control(get_responses()): File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 416, in __next__ return self._next() File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 706, in _next raise self grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with: status = StatusCode.RESOURCE_EXHAUSTED details = "Received message larger than max (4847139 vs. 4194304)" debug_error_string = "{"created":"@1593023849.325990420","description":"Received message larger than max (4847139 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":188,"grpc_status":8}"
      

       

      This can be fixed by modifying the options parameter to allow any size grpc message to be sent or received.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kshivvy Keshav
            kshivvy Keshav
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1.5h
                1.5h

                Issue deployment