[HOP-3767] Problems using a long-lived docker container with the web service

• Type: Bug
• Status: Resolved
• Priority: High
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: Containers, docker, Hop Server
• Labels: pull-request-available
• Assignee: Hans Van Akelyen
• Reporter: Fiction Becomes Fact
• Created: 17/Feb/22 11:29
• Updated: 23/Feb/22 10:21
• Resolved: 23/Feb/22 10:21

Description

I would like to use a long-lived docker container to run a web service. This with a reference to a project and environment.

1. Within the "Web Service" configuration the filename on the server is "${PROJECT_HOME}/webservice-test.hpl".

2. I copy the project to the mapping for "/files/projects/hop-server-test". Then the metadata is in "/files/projects/hop-server-test/metadata". I copy the "default" project to "/files/projects/default" just to be sure.

3. I copy the environment config file to the mapping for "/files/env/hop-server-test-development-config.json".

4. Example run script, based on this issue:

docker run -d \
 --name=hop \
 --hostname=hop \
 -p 8182:8182 \
 -v /mnt/fileserver/hop:/files \
 -v $PWD/tmp:/tmp \
 -e TZ=Europe/Amsterdam \
 -e HOP_SERVER_PORT="8182" \
 -e HOP_SERVER_HOSTNAME="0.0.0.0" \
 -e HOP_SERVER_USER="admin" \
 -e HOP_SERVER_PASS="admin" \
 -e HOP_PROJECT_NAME="Hop Server Test" \
 -e HOP_PROJECT_FOLDER="/files/projects/hop-server-test" \
 -e HOP_ENVIRONMENT_NAME="development-config" \
 -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS="/files/env/hop-server-test-development-config.json" \
 -e HOP_SERVER_METADATA_FOLDER="/files/projects/hop-server-test/metadata" \
 -e HOP_SHARED_JDBC_FOLDER="/files/jdbc" \
 -e HOP_LOG_LEVEL="Debug" \
 -e HOP_LOG_PATH="/files/log/hop.err.log" \
 --restart unless-stopped \
 apache/hop

 
5. The Docker logs are as follows:

2022/02/17 10:33:43 - Running the entrypoint script with PID 6
2022/02/17 10:33:43 - The project folder for Hop Server Test is set to: /files/projects/hop-server-test
2022/02/17 10:33:43 - The specified project folder exists
2022/02/17 10:33:43 - Registering project Hop Server Test in the Hop container configuration
2022/02/17 10:33:43 - /opt/hop/hop/hop-conf.sh --project=Hop Server Test --project-create --project-home='/files/projects/hop-server-test' --project-config-file='project-config.json'
Creating project 'Hop Server Test'
Project 'Hop Server Test' was created for home folder : /files/projects/hop-server-test
Configuration file for project 'Hop Server Test' was saved to : file:/files/projects/hop-server-test/project-config.json
2022/02/17 10:33:47 - Registering environment development-config in the Hop container configuration
2022/02/17 10:33:47 - /opt/hop/hop/hop-conf.sh --environment-create --environment=development-config --environment-project=Hop Server Test --environment-config-files='/files/env/hop-server-test-development-config.json'
Creating environment 'development-config'
Environment 'development-config' was created in Hop configuration file /opt/hop/hop/config/hop-config.json
Found existing environment configuration file: /files/env/hop-server-test-development-config.json
  development-config
    Purpose: Apache Hop docker container
    Project name: Hop Server Test
      Config file: /files/env/hop-server-test-development-config.json
2022/02/17 10:33:50 - Writing a hop-server config file to /tmp/hop-server.xml
2022/02/17 10:33:50 - The server metadata is in folder: /files/projects/hop-server-test/metadata
2022/02/17 10:33:50 - Starting a hop-server on port 8182
2022/02/17 11:33:53 - HopServer - Start of Hop Server
2022/02/17 11:33:53 - HopServer - Referencing environment 'development-config' for project Hop Server Test' in Apache Hop docker container
2022/02/17 11:33:53 - HopServer - Enabling project 'Hop Server Test'
2022/02/17 11:33:53 - HopServer - Installing timer to purge stale objects after 1440 minutes.
2022/02/17 11:33:53 - HopServer - Created listener for webserver @ address : localhost:8080

 

PROBLEM #1

The documentation is not clear about using projects and environments for a web service using a long-lived docker container. The web service documentation might also be supplemented.

 

PROBLEM #2

The container starts up but creates a listener for webserver @ address : localhost:8080 (see the docker logs) and not at 0.0.0.0 with port 8182 as it should work. Within tmp\hop-server.xml are the correct settings:

<hop-server-config>
  <hop-server>
    <name>Hop Server</name>
    <hostname>0.0.0.0</hostname>
    <port>8182</port>
    <username>admin</username>
    <password>admin</password>
  </hop-server>
  <metadata_folder>/files/projects/hop-server-test/metadata</metadata_folder>
</hop-server-config>

 

PROBLEM #3

The container cannot be restarted after it has been created. Commands: docker stop hop && docker start hop. The container returns the following error:

2022/02/17 10:47:38 - Stopping entrypoint script with 6
2022/02/17 10:47:44 - Running the entrypoint script with PID 8
2022/02/17 10:47:44 - The project folder for Hop Server Test is set to: /files/projects/hop-server-test
2022/02/17 10:47:44 - The specified project folder exists
2022/02/17 10:47:44 - Registering project Hop Server Test in the Hop container configuration
2022/02/17 10:47:44 - /opt/hop/hop/hop-conf.sh --project=Hop Server Test --project-create --project-home='/files/projects/hop-server-test' --project-config-file='project-config.json'
Creating project 'Hop Server Test'
Error found during execution!
picocli.CommandLine$ExecutionException: There was an error handling options
        at org.apache.hop.config.HopConfig.run(HopConfig.java:88)
        at org.apache.hop.config.HopConfig.main(HopConfig.java:160)
Caused by: org.apache.hop.core.exception.HopException:
Error handling environment configuration options
Project 'Hop Server Test' already exists.

        at org.apache.hop.projects.project.ManageProjectsOptionPlugin.handleOption(ManageProjectsOptionPlugin.java:163)
        at org.apache.hop.config.HopConfig.run(HopConfig.java:79)
        ... 1 more
Caused by: org.apache.hop.core.exception.HopException:
Project 'Hop Server Test' already exists.
        at org.apache.hop.projects.project.ManageProjectsOptionPlugin.createProject(ManageProjectsOptionPlugin.java:327)
        at org.apache.hop.projects.project.ManageProjectsOptionPlugin.handleOption(ManageProjectsOptionPlugin.java:146)
        ... 2 more

{+}
PROBLEM #4
{+}If no project is used, but only a pipeline with a metadata folder, then the web service works. But because the project is missing the environment does not work (I think) and environment variables are not replaced in the output of the web service.
Within the "Web Service" configuration the filename on the server is "/files/transforms/webservice-test.hpl".
Example run script:

docker run -d \
 --name=hop \
 --hostname=hop \
 -p 8182:8182 \
 -v /mnt/fileserver/hop:/files \
 -e TZ=Europe/Amsterdam \
 -e HOP_SERVER_PORT=8182 \
 -e HOP_SERVER_HOSTNAME=0.0.0.0 \
 -e HOP_SERVER_USER=admin \
 -e HOP_SERVER_PASS=admin \
 -e HOP_ENVIRONMENT_NAME="development-config" \
 -e HOP_ENVIRONMENT_CONFIG_FILE_NAME_PATHS="/files/env/hop-server-test-development-config.json" \
 -e HOP_SERVER_METADATA_FOLDER=/files/metadata \
 -e HOP_SHARED_JDBC_FOLDER=/files/jdbc \
 -e HOP_LOG_PATH=/files/log/hop.err.log \
 --restart unless-stopped \
 apache/hop

Issue Links

• is related to: HOP-3667 Documentation: Tutorial for HOP Docker and Web services
• links to: GitHub Pull Request #1385

Sub-Tasks

• Docker Hop server: detect if project or environment already exists: Resolved
• Docker Hop server: spaces in object names cause server-xml to not be loaded: Resolved