[BEAM-5190] Python pipeline options are not picked correctly by PortableRunner

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.8.0
• Component/s: sdk-py-harness
• Assignee: Ankur Goenka
• Reporter: Ankur Goenka
• Created: 22/Aug/18 07:31
• Updated: 16/May/20 13:59
• Resolved: 03/Oct/18 00:43

Description

Python SDK worker is deserializing the pipeline options to dictionary instead of PipelineOptions

Sample log

[grpc-default-executor-2] INFO sdk_worker_main.main - Python sdk harness started with pipeline_options: {u'beam:option:flink_master:v1': u'[auto]', u'beam:option:streaming:v1': False, u'beam:option:experiments:v1': [u'beam_fn_api', u'worker_threads=50'], u'beam:option:dry_run:v1': False, u'beam:option:runner:v1': None, u'beam:option:profile_memory:v1': False, u'beam:option:runtime_type_check:v1': False, u'beam:option:region:v1': u'us-central1', u'beam:option:options_id:v1': 1, u'beam:option:no_auth:v1': False, u'beam:option:dataflow_endpoint:v1': u'https://dataflow.googleapis.com', u'beam:option:sdk_location:v1': u'/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/dist/apache-beam-2.7.0.dev0.tar.gz', u'beam:option:direct_runner_use_stacked_bundle:v1': True, u'beam:option:save_main_session:v1': True, u'beam:option:type_check_strictness:v1': u'DEFAULT_TO_ANY', u'beam:option:profile_cpu:v1': False, u'beam:option:job_endpoint:v1': u'localhost:8099', u'beam:option:job_name:v1': u'BeamApp-goenka-0822071645-48ae1008', u'beam:option:temp_location:v1': u'gs://clouddfe-goenka/tmp/', u'beam:option:app_name:v1': None, u'beam:option:project:v1': u'google.com:clouddfe', u'beam:option:pipeline_type_check:v1': True, u'beam:option:staging_location:v1': u'gs://clouddfe-goenka/tmp/staging'} 

 

Issue Links

• links to: GitHub Pull Request #6267