Uploaded image for project: 'Apache Hop (incubating)'
  1. Apache Hop (incubating)
  2. HOP-2070

Pipeline run config does not pick up environment variable

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.60
    • Component/s: None
    • Labels:

      Description

      I've got several variables (e.g. VAR_GCP_DATAFLOW_TEMP_LOCATION) defined in my `hop-beam-examples-dev` environment definition. I reference these variables in my pipeline run configuration `beam-dataflow`, however, for some reason they are not picked up. The code is available here. Please see below for the run command I issued and the error message:

      ```
      export HOP_BINARY_PATH=/Users/diethardsteiner/apps/hop (master)hop-beam-examples
      ${HOP_BINARY_PATH}/hop-run.sh \
      --file=pipelines-and-workflows/transform-event-data-to-json.hpl \
      --project=hop-beam-examples \
      --environment=hop-beam-examples-dev \
      --runconfig=beam-dataflow \
      --level=Basic \
      --parameters=PARAM_INPUT_DATASET_FIELD_ENCLOSURE=,PARAM_INPUT_DATASET_FIELD_SEPARATOR=",",PARAM_INPUT_DATASET_FILE_PATH=gs://hop-beam-storage/input/event-data.csv,PARAM_OUTPUT_DATASET_FIELD_ENCLOSURE=,PARAM_OUTPUT_DATASET_FIELD_SEPARATOR=,PARAM_OUTPUT_DATASET_FILE_PATH=gs://hop-beam-storage/output,PARAM_OUTPUT_DATASET_FILE_PREFIX=events,PARAM_OUTPUT_DATASET_SUFFIX=.csv \
      2>&1 | tee /tmp/transform-event-data-to-json.hpl.err.log
      2020/08/25 08:18:15 - HopRun - Referencing environment 'hop-beam-examples-dev' for project hop-beam-examples' in development
      2020/08/25 08:18:15 - HopRun - Enabling project 'hop-beam-examples'
      2020/08/25 08:18:15 - HopRun - Relative path filename specified: /Users/diethardsteiner/git/hop-beam-examples/project/pipelines-and-workflows/transform-event-data-to-json.hpl
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/Users/diethardsteiner/apps/hop/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/Users/diethardsteiner/apps/hop/plugins/engines/beam/lib/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      Error found during execution!
      picocli.CommandLine$ExecutionException: There was an error during execution of file 'pipelines-and-workflows/transform-event-data-to-json.hpl'
      at org.apache.hop.run.HopRun.run(HopRun.java:124)
      at org.apache.hop.run.HopRun.main(HopRun.java:686)
      Caused by: picocli.CommandLine$ExecutionException: There was an error during execution of pipeline 'pipelines-and-workflows/transform-event-data-to-json.hpl'
      at org.apache.hop.run.HopRun.runPipeline(HopRun.java:195)
      at org.apache.hop.run.HopRun.run(HopRun.java:116)
      ... 1 more
      Caused by: picocli.CommandLine$ExecutionException: Error running pipeline locally
      at org.apache.hop.run.HopRun.runPipeline(HopRun.java:231)
      at org.apache.hop.run.HopRun.runPipeline(HopRun.java:192)
      ... 2 more
      Caused by: org.apache.hop.core.exception.HopException:
      Error preparing remote pipeline
      Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)

      at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:253)
      at org.apache.hop.run.HopRun.runPipeline(HopRun.java:227)
      ... 3 more
      Caused by: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
      at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
      at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
      at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
      at org.apache.beam.sdk.Pipeline.create(Pipeline.java:149)
      at org.apache.hop.beam.pipeline.HopPipelineMetaToBeamPipelineConverter.createPipeline(HopPipelineMetaToBeamPipelineConverter.java:216)
      at org.apache.hop.beam.engines.BeamPipelineEngine.prepareExecution(BeamPipelineEngine.java:225)
      ... 4 more
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
      ... 9 more
      Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions
      at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:271)
      ... 14 more
      Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, ${VAR_GCP_DATAFLOW_TEMP_LOCATION}.
      at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:317)
      at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:294)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
      at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:158)
      at com.sun.proxy.$Proxy23.getGcpTempLocation(Unknown Source)
      at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:269)
      ... 14 more
      Caused by: java.lang.IllegalArgumentException: Expected a valid 'gs://' path but was given '${VAR_GCP_DATAFLOW_TEMP_LOCATION}'
      at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:96)
      at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPath(GcsPathValidator.java:67)
      at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:52)
      at org.apache.beam.sdk.extensions.gcp.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:314)
      ... 20 more
      Caused by: java.lang.IllegalArgumentException: Invalid GCS URI: ${VAR_GCP_DATAFLOW_TEMP_LOCATION}
      at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions.checkArgument(Preconditions.java:216)
      at org.apache.beam.sdk.extensions.gcp.util.gcsfs.GcsPath.fromUri(GcsPath.java:116)
      at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:93)
      ... 23 more
      ```

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mxm Maximilian Michels
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: