Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3585

Python dataflow job fails with 2.3.0 RC1, due to missing worker image

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P0
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: examples-python
    • Labels:
      None

      Description

      The dataflow python jobs currently fail due to a missing docker image when using 2.3.0 RC1. This is not a bug in the SDK, the worker image needs to be published by google. I will be coordinating the worker image publication.

      1. Update to your own project and bucket.

      GCS_BUCKET=my-cloud-storage-bucket

      GCP_PROJECT=my-cloud-project

      virtualenv env

      . env/bin/activate

      wget https://dist.apache.org/repos/dist/dev/beam/2.3.0/apache-beam-2.3.0-python.zip

      pip install apache-beam-2.3.0-python.zip[gcp]

      python m apache_beam.examples.wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt --output gs://${GCS_BUCKET}/counts  -runner DataflowRunner --project ${GCP_PROJECT} --temp_location gs://${GCS_BUCKET}/tmp --sdk_location apache-beam-2.3.0-python.zip

       

      Dataflow logs contain:

      E  Handler for GET /v1.27/images/dataflow.gcr.io/v1beta3/python:2.3.0/json returned error: No such image: dataflow.gcr.io/v1beta3/python:2.3.0 
      E  container start failed: ImagePullBackOff: Back-off pulling image "dataflow.gcr.io/v1beta3/python:2.3.0"

        Attachments

          Activity

            People

            • Assignee:
              alanmyrvold Alan Myrvold
              Reporter:
              alanmyrvold Alan Myrvold
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: