[BEAM-5442] PortableRunner swallows custom options for Runner

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.11.0
• Component/s: sdk-java-core, sdk-py-core
• Labels: Done portability portability-flink
• Assignee: Thomas Weise
• Reporter: Maximilian Michels
• Created: 20/Sep/18 15:18
• Updated: 06/Oct/20 21:53
• Resolved: 29/Jan/19 20:01

Description

The PortableRunner doesn't pass custom PipelineOptions to the executing Runner.

Example: --parallelism=4 won't be forwarded to the FlinkRunner.
(The option is just removed during proto translation without any warning)

We should allow some form of customization through the options, even for the PortableRunner.

Issue Links

• relates to: BEAM-6549 Make use of DescribePipelineOptionsRequest in Go SDK
• relates to: BEAM-1318 PipelineOptions should warn if there are unused options
• relates to: BEAM-6531 Namespace for pipeline options
• relates to: BEAM-6537 Separate runner and SDK options
• links to: GitHub Pull Request #6447
• links to: GitHub Pull Request #6557
• links to: GitHub Pull Request #6584
• links to: GitHub Pull Request #6589
• links to: GitHub Pull Request #6600
• links to: GitHub Pull Request #6683
• links to: GitHub Pull Request #7117
• links to: GitHub Pull Request #7564
• links to: GitHub Pull Request #7597