[BEAM-3645] Support multi-process execution on the FnApiRunner

• Type: Improvement
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.2.0, 2.3.0
• Fix Version/s: 2.15.0
• Component/s: sdk-py-core
• Assignee: Hannah Jiang
• Reporter: Charles Chen
• Created: 07/Feb/18 23:20
• Updated: 13/Apr/23 10:56
• Resolved: 31/Jul/19 17:44

Description

https://issues.apache.org/jira/browse/BEAM-3644 gave us a 15x performance gain over the previous DirectRunner.  We can do even better in multi-core environments by supporting multi-process execution in the FnApiRunner, to scale past Python GIL limitations.

Issue Links

• is blocked by: BEAM-7873 FnApi with Subprocess runner hangs frequently when running with multi workers with py2
• is blocked by: BEAM-7874 FnApi only supports up to 10 workers
• relates to: BEAM-7997 Create a single docker container for a pipeline instead of creating one for each worker.
• relates to: BEAM-7888 Test Multi Process Direct Runner With Largish Data
• links to: GitHub Pull Request #8769
• links to: GitHub Pull Request #8872
• links to: GitHub Pull Request #8979
• links to: GitHub Pull Request #9002
• links to: GitHub Pull Request #9051

Sub-Tasks

• Split GroupingBuffer to N pieces: Resolved
• Multiplexing chunked GroupingBuffer: Resolved
• Make it easy to change between multi-process and multi-thread mode for Python Direct runners: Resolved