[BEAM-3585] Python dataflow job fails with 2.3.0 RC1, due to missing worker image

• Type: Bug
• Status: Resolved
• Priority: P0
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: examples-python
• Assignee: Alan Myrvold
• Reporter: Alan Myrvold
• Created: 31/Jan/18 21:14
• Updated: 16/May/20 14:21
• Resolved: 01/Feb/18 04:21

Description

The dataflow python jobs currently fail due to a missing docker image when using 2.3.0 RC1. This is not a bug in the SDK, the worker image needs to be published by google. I will be coordinating the worker image publication.

1. Update to your own project and bucket.

GCS_BUCKET=my-cloud-storage-bucket

GCP_PROJECT=my-cloud-project

virtualenv env

. env/bin/activate

wget https://dist.apache.org/repos/dist/dev/beam/2.3.0/apache-beam-2.3.0-python.zip

pip install apache-beam-2.3.0-python.zip[gcp]

python m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://${GCS_BUCKET}/counts  -runner DataflowRunner --project ${GCP_PROJECT} --temp_location gs://${GCS_BUCKET}/tmp --sdk_location apache-beam-2.3.0-python.zip

 

Dataflow logs contain:

E  Handler for GET /v1.27/images/dataflow.gcr.io/v1beta3/python:2.3.0/json returned error: No such image: dataflow.gcr.io/v1beta3/python:2.3.0 
E  container start failed: ImagePullBackOff: Back-off pulling image "dataflow.gcr.io/v1beta3/python:2.3.0"