Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
I've got several variables (e.g. VAR_GCP_DATAFLOW_TEMP_LOCATION) defined in my `hop-beam-examples-dev` environment definition. I reference these variables in my pipeline run configuration `beam-dataflow`, however, for some reason they are not picked up. The code is available here. Please see below for the run command I issued and the error message:
```
export HOP_BINARY_PATH=/Users/diethardsteiner/apps/hop (master)hop-beam-examples
${HOP_BINARY_PATH}/hop-run.sh \
--file=pipelines-and-workflows/transform-event-data-to-json.hpl \
--project=hop-beam-examples \
--environment=hop-beam-examples-dev \
--runconfig=beam-dataflow \
--level=Basic \
--parameters=PARAM_INPUT_DATASET_FIELD_ENCLOSURE=,PARAM_INPUT_DATASET_FIELD_SEPARATOR=",",PARAM_INPUT_DATASET_FILE_PATH=gs://hop-beam-storage/input/event-data.csv,PARAM_OUTPUT_DATASET_FIELD_ENCLOSURE=,PARAM_OUTPUT_DATASET_FIELD_SEPARATOR=,PARAM_OUTPUT_DATASET_FILE_PATH=gs://hop-beam-storage/output,PARAM_OUTPUT_DATASET_FILE_PREFIX=events,PARAM_OUTPUT_DATASET_SUFFIX=.csv \
2>&1 | tee /tmp/transform-event-data-to-json.hpl.err.log
2020/08/25 08:18:15 - HopRun - Referencing environment 'hop-beam-examples-dev' for project hop-beam-examples' in development
2020/08/25 08:18:15 - HopRun - Enabling project 'hop-beam-examples'
2020/08/25 08:18:15 - HopRun - Relative path filename specified: /Users/diethardsteiner/git/hop-beam-examples/project/pipelines-and-workflows/transform-event-data-to-json.hpl
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/diethardsteiner/apps/hop/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/diethardsteiner/apps/hop/plugins/engines/beam/lib/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Error found during execution!
picocli.CommandLine$ExecutionException: There was an error during execution of file 'pipelines-and-workflows/transform-event-data-to-json.hpl'
at org.apache.hop.run.HopRun.run(HopRun.java:124)
at org.apache.hop.run.HopRun.main(HopRun.java:686)
Caused by: picocli.CommandLine$ExecutionException: There was an error during execution of pipeline 'pipelines-and-workflows/transform-event-data-to-json.hpl'
at org.apache.hop.run.HopRun.runPipeline(HopRun.java:195)
at org.apache.hop.run.HopRun.run(HopRun.java:116)
... 1 more
Caused by: picocli.CommandLine$ExecutionException: Error running pipeline locally
at org.apache.hop.run.HopRun.runPipeline(HopRun.java:231)
at org.apache.hop.run.HopRun.runPipeline(HopRun.java:192)
... 2 more
Caused by: org.apache.hop.core.exception.HopException:
Error preparing remote pipeline
Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:253)
at org.apache.hop.run.HopRun.runPipeline(HopRun.java:227)
... 3 more
Caused by: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:149)
at org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.createPipeline(HopPipelineMetaToBeamPipelineConverter.java:216)
at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:225)
... 4 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
... 9 more
Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:271)
... 14 more
Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, ${VAR_GCP_DATAFLOW_TEMP_LOCATION}.
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:317)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:294)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:158)
at com.sun.proxy.$Proxy23.getGcpTempLocation(Unknown Source)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:269)
... 14 more
Caused by: java.lang.IllegalArgumentException: Expected a valid 'gs://' path but was given '${VAR_GCP_DATAFLOW_TEMP_LOCATION}'
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:96)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPath(GcsPathValidator.java:67)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:52)
at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:314)
... 20 more
Caused by: java.lang.IllegalArgumentException: Invalid GCS URI: ${VAR_GCP_DATAFLOW_TEMP_LOCATION}
at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)
at org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath.fromUri(GcsPath.java:116)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:93)
... 23 more
```