[BEAM-10671] Add environment configuration fields as first-class pipeline options

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.29.0
• Component/s: sdk-java-harness, sdk-py-harness
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 10/Aug/20 21:45
• Updated: 02/Apr/21 20:42
• Resolved: 02/Apr/21 20:37

Description

The pipeline option --environment_config has completely different usages depending on the value of --environment_type. This is confusing for the user and hard to check. Additionally, --environment_config is a JSON blob for --environment_type=PROCESS. This JSON blob is a pain to escape and pass around compared to a collection of flat strings.

We should replace --environment_config with first-class / top-level pipeline options for each environment type:

DOCKER

--environment_container_image

PROCESS

--environment_os

--environment_architecture

--environment_variables

EXTERNAL

--environment_service_address

LOOPBACK

(none)

This way we can validate that the user is configuring these options correctly (ie give a warning or error if they use options that do not apply to their chosen --environment_type).

We can deprecate the --environment_config option, logging a warning until removing this option altogether in a future Beam release.

https://beam.apache.org/documentation/runtime/sdk-harness-config/

Issue Links

• links to: GitHub Pull Request #12576
• links to: GitHub Pull Request #13908