[BEAM-10275] sdk_worker_main.py eagerly parses pipeline options

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-py-harness
• Assignee: Brian Hulette
• Reporter: Brian Hulette
• Created: 18/Jun/20 15:40
• Updated: 15/May/21 17:58
• Resolved: 15/May/21 17:58

Description

sdk_worker_main.py eagerly parses pipeline options beause of the call to get_all_options here: https://github.com/apache/beam/blob/61b665640d6c0f91751bba59782c0ac6aceacba6/sdks/python/apache_beam/runners/worker/sdk_worker_main.py#L138

This can cause the worker to crash if any option that can't be read at execution is time is used, even if we don't need to access it at execution time (e.g. json.loads arguments, described in BEAM-10274)

Issue Links

• is related to: BEAM-10274 Python SDK can't parse type=json.loads pipeline options at execution time
• links to: GitHub Pull Request #12032