[BEAM-7874] FnApi only supports up to 10 workers

• Type: Bug
• Status: Resolved
• Priority: P0
• Resolution: Fixed
• Fix Version/s: 2.15.0
• Component/s: sdk-py-core
• Assignee: Hannah Jiang
• Reporter: Hannah Jiang
• Created: 01/Aug/19 18:27
• Updated: 16/May/20 14:20
• Resolved: 12/Aug/19 23:07

Description

Because max_workers of grpc servers are hardcoded to 10, it only supports up to 10 workers, and if we pass more direct_num_workers greater than 10, pipeline hangs, because not all workers get connected to the runner.

https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner.py#L1141

Issue Links

• blocks: BEAM-3645 Support multi-process execution on the FnApiRunner
• links to: GitHub Pull Request #9218
• links to: GitHub Pull Request #9324