[MINIFICPP-1355] Investigate and fix the initialization of ExecutePythonProcessor

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 1.0.0, 0.12.0
• Labels: MiNiFi-CPP-Hygiene
• Assignee: Gábor Gyimesi
• Reporter: Adam Hunyadi
• Created: 04/Sep/20 14:09
• Updated: 03/Jan/22 09:35
• Resolved: 26/Aug/21 08:07

Description

Acceptance criteria:

• GIVEN a flow set up as highlighted in blue below
• WHEN the flow is with a python script set to add a new attribute to a flow file
• THEN no error is produced and the newly added attribute is logged in LogAttribute

Example script

def describe(processor):
    processor.setDescription("Adds an attribute to your flow files")

def onInitialize(processor):
  processor.setSupportsDynamicProperties()

def onTrigger(context, session):
  flow_file = session.get()
  if flow_file is not None:
    flow_file.addAttribute("Python attribute","attributevalue")
    session.transfer(flow_file, REL_SUCCESS)

Background:

Currently, even though the tests for ExecutePythonProcessor are passing, if I were to try and load up a configuration that contains an ExecutePythonProcessor, it fails due to trying to load an incorrect script file.

Sample flow:
GenerateFlowFile (success)> ExecutePythonProcessor (success,failure)> LogAttribute

When trying to check in debugger, it seems like the processors script file is always replaced with an incorrect one, and the processor fails to start.

This is how it is set:

Trace of where the property is overridden

ConfigurableComponent::setProperty()
std::shared_ptr<core::CoreComponent> create()
ClassLoader::instantiate()
PythonCreator::configure() <- here the first element of classpaths_ is read to overwrite the config
FlowController::initializeExternalComponents()

When trying to perform the same thing on the 0.7.0 release version, the startup already shows some kind of errors, although they seem different:

Error log

[2020-09-04 15:49:53.424] [org::apache::nifi::minifi::python::processors::ExecutePythonProcessor] [error] Caught Exception ModuleNotFoundError: No module named 'google'
At:
 /Users/adamhunyadi/Documents/Projects/integration_tests/minifi_agent_02/build/nifi-minifi-cpp-0.7.0/minifi-python//google/SentimentAnalyzer.py(28): <module>
[2020-09-04 15:49:53.424] [org::apache::nifi::minifi::python::PythonCreator] [warning] Cannot load SentimentAnalyzer because of ModuleNotFoundError: No module named 'google'
At:
 /Users/adamhunyadi/Documents/Projects/integration_tests/minifi_agent_02/build/nifi-minifi-cpp-0.7.0/minifi-python//google/SentimentAnalyzer.py(28): <module>
[2020-09-04 15:49:53.424] [org::apache::nifi::minifi::python::processors::ExecutePythonProcessor] [error] Caught Exception ModuleNotFoundError: No module named 'vaderSentiment'
At:
 /Users/adamhunyadi/Documents/Projects/integration_tests/minifi_agent_02/build/nifi-minifi-cpp-0.7.0/minifi-python//examples/SentimentAnalysis.py(17): <module>
[2020-09-04 15:49:53.424] [org::apache::nifi::minifi::python::PythonCreator] [warning] Cannot load SentimentAnalysis because of ModuleNotFoundError: No module named 'vaderSentiment'
At:
 /Users/adamhunyadi/Documents/Projects/integration_tests/minifi_agent_02/build/nifi-minifi-cpp-0.7.0/minifi-python//examples/SentimentAnalysis.py(17): <module>

Proposal:

One should investigate and fix the error.