[BEAM-3584] Java dataflow job fails with 2.3.0 RC1, due to missing worker image

• Type: Bug
• Status: Resolved
• Priority: P0
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0
• Component/s: examples-java
• Assignee: Alan Myrvold
• Reporter: Alan Myrvold
• Created: 31/Jan/18 21:14
• Updated: 16/May/20 14:21
• Resolved: 03/Feb/18 01:24

Description

Dataflow java jobs currently fail due to a missing docker image when using 2.3.0 RC1. This is not a bug in the SDK, the worker image needs to be published by google. I will be coordinating the worker image publication.

1. Update to your own project and bucket.

GCS_BUCKET=my-cloud-storage-bucket

GCP_PROJECT=my-cloud-project

mkdir -p ~/try2.3.0/.m2/repository

cd ~/try2.3.0/.m2/repository

wget https://repository.apache.org/content/repositories/orgapachebeam-1026/archetype-catalog.xml

cd ~/try2.3.0

echo "<settings><localRepository>$PWD/.m2/repository</localRepository><profiles><profile><id>testrel</id><repositories><repository><id>testrel</id><url>https://repository.apache.org/content/repositories/orgapachebeam-1026</url></repository></repositories></profile></profiles></settings>" > settings.xml

mvn archetype:generate -DarchetypeCatalog=local  -DarchetypeGroupId=org.apache.beam
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples       -DarchetypeVersion=2.3.0      -DgroupId=org.example       -DartifactId=word-count-beam       -Dversion="0.1"       -Dpackage=org.apache.beam.examples -s settings.xml

1. answer yes

cd word-count-beam

mvn compile exec:java Dexec.mainClass=org.apache.beam.examples.WordCount        -Dexec.args="runner=DataflowRunner --project=alan-jenkins-test  gcpTempLocation=gs://${GCS_BUCKET}/tmp  -inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://${GCP_PROJECT}e/counts"   -s ../settings.xml -Ptestrel,dataflow-runner

 

Dataflow logs contain:

E  Handler for GET /v1.27/images/dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0/json returned error: No such image: dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0
E  PullImage "dataflow.gcr.io/v1beta3/beam-java-batch:beam-2.3.0" from image service failed: rpc error: code = 2 desc = Error: Status 405 trying to pull repository v1beta3/beam-java-batch: "v1 Registry API is disabled. If you are not explicitly using the v1 Registry API, it is possible your v2 image could not be found. Verify that your image is available, or retry with `dockerd --disable-legacy-registry`. See https://cloud.google.com/container-registry/docs/support/deprecation-notices"