Details
-
New Feature
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.21.0
-
None
-
None
Description
Hi,
I'm running Apache NiFi in Docker. To make it stateful I had to apply the following changes:
nifi.properties
nifi.database.directory=/opt/nifi/repositories/database_repository nifi.flowfile.repository.directory=/opt/nifi/repositories/flowfile_repository nifi.content.repository.directory.default=/opt/nifi/repositories/content_repository nifi.provenance.repository.directory.default=/opt/nifi/repositories/provenance_repository nifi.status.repository.questdb.persist.location=/opt/nifi/repositories/status_repository nifi.flow.configuration.file=/opt/nifi/repositories/configuration_repository/flow.xml.gz nifi.flow.configuration.json.file=/opt/nifi/repositories/configuration_repository/flow.json.gz nifi.flow.configuration.archive.dir=/opt/nifi/repositories/configuration_repository/archive/
state-management.xml
<property name="Directory">/opt/nifi/repositories/state_repository/local</property>
It would be really convenient to set these values using environment variables, however the current NiFi Docker image has no support for this.
As a temporary solution I have overwritten the image's default entrypoint and used a self-written script:
#!/bin/bash scripts_dir='/opt/nifi/scripts' [ -f "$${scripts_dir}/common.sh" ] && . "$${scripts_dir}/common.sh" prop_replace 'nifi.database.directory' "/opt/nifi/repositories/database_repository" prop_replace 'nifi.flowfile.repository.directory' "/opt/nifi/repositories/flowfile_repository" prop_replace 'nifi.content.repository.directory.default' "/opt/nifi/repositories/content_repository" prop_replace 'nifi.provenance.repository.directory.default' "/opt/nifi/repositories/provenance_repository" prop_replace 'nifi.status.repository.questdb.persist.location' "/opt/nifi/repositories/status_repository" prop_replace 'nifi.flow.configuration.file' "/opt/nifi/repositories/configuration_repository/flow.xml.gz" prop_replace 'nifi.flow.configuration.json.file' "/opt/nifi/repositories/configuration_repository/flow.json.gz" prop_replace 'nifi.flow.configuration.archive.dir' "/opt/nifi/repositories/configuration_repository/archive/" echo "Hacking nifi.properties for stateful operation completed" sed -i -e 's|<property name="Directory">./state/local</property>|<property name="Directory">/opt/nifi/repositories/state_repository/local</property>|' $${NIFI_HOME}/conf/state-management.xml echo "Hacking state-management.xml for stateful operation completed" echo "Calling start.sh" . /opt/nifi/scripts/start.sh
Proposed solution
I would like to craete a new shell-script (update_stateful_paths.sh) for setting the required variables:
#!/bin/sh -e # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. prop_replace 'nifi.database.directory' "${NIFI_DATABASE_DIRECTORY:-./database_repository}" prop_replace 'nifi.flowfile.repository.directory' "${NIFI_FLOWFILE_REPOSITORY_DIRECTORY:-./flowfile_repository}" prop_replace 'nifi.content.repository.directory.default' "${NIFI_CONTENT_REPOSITORY_DIRECTORY_DEFAULT:-./content_repository}" prop_replace 'nifi.provenance.repository.directory.default' "${NIFI_PROVENANCE_REPOSITORY_DIRECTORY_DEFAULT:-./provenance_repository}" prop_replace 'nifi.status.repository.questdb.persist.location' "${NIFI_STATUS_REPOSITORY_QUESTDB_PERSIST_LOCATION:-./status_repository}" prop_replace 'nifi.flow.configuration.file' "${NIFI_FLOW_CONFIGURATION_FILE:-./conf/flow.xml.gz}" prop_replace 'nifi.flow.configuration.json.file' "${NIFI_FLOW_CONFIGURATION_JSON_FILE:-./conf/flow.json.gz}" prop_replace 'nifi.flow.configuration.archive.dir' "${NIFI_FLOW_CONFIGURATION_ARCHIVE_DIR:-./conf/archive/}" sed -i -e "s|<property name=\"Directory\">./state/local</property>|<property name=\"Directory\">${NIFI_STATE_REPOSITORY_DIRECTORY:-./state/local}</property>|" ${NIFI_HOME}/conf/state-management.xml
The start.sh would call this new script:
. "${scripts_dir}/secure.sh" . "${scripts_dir}/update_oidc_properties.sh" ;; esac # ... # NEW LINE: . "${scripts_dir}/update_stateful_paths.sh" # ... "${NIFI_HOME}/bin/nifi.sh" run & nifi_pid="$!" tail -F --pid=${nifi_pid} "${NIFI_HOME}/logs/nifi-app.log" &
I have tested it with the following env file:
NIFI_DATABASE_DIRECTORY=/opt/nifi/repositories/db NIFI_FLOWFILE_REPOSITORY_DIRECTORY=/opt/nifi/repositories/flow NIFI_CONTENT_REPOSITORY_DIRECTORY_DEFAULT=/opt/nifi/repositories/content NIFI_PROVENANCE_REPOSITORY_DIRECTORY_DEFAULT=/opt/nifi/repositories/prov NIFI_STATUS_REPOSITORY_QUESTDB_PERSIST_LOCATION=/opt/nifi/repositories/status NIFI_FLOW_CONFIGURATION_FILE=/opt/nifi/repositories/conf/flow.xml.gz NIFI_FLOW_CONFIGURATION_JSON_FILE=/opt/nifi/repositories/conf/flow.json.gz NIFI_FLOW_CONFIGURATION_ARCHIVE_DIR=/opt/nifi/repositories/conf/archive NIFI_STATE_REPOSITORY_DIRECTORY=/opt/nifi/repositories/state
And then running Docker:
docker run -it \ --env-file /Users/tneumer/Downloads/nifi_env_file \ -v /Users/tneumer/Downloads/nifi_temp:/opt/nifi/repositories \ -v /Users/tneumer/temp/nifi/nifi-docker/dockerhub/sh/start.sh:/opt/nifi/scripts/start.sh \ -v /Users/tneumer/temp/nifi/nifi-docker/dockerhub/sh/update_stateful_paths.sh:/opt/nifi/scripts/update_stateful_paths.sh \ apache/nifi:1.21.0 bash
If you approve, I would create a pull-request with the proposed solution!
Kind regards,
Tamas