[NIFI-8684] sensitive property not working for InvokeScriptedProcessor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.2
• Fix Version/s: 1.15.0
• Component/s: Extensions
• Labels: InvokeScriptedProcessor property sensitive
• Assignee: Matt Burgess
• Reporter: Jul Tomten
• Created: 11/Jun/21 11:32
• Updated: 28/Oct/21 20:53
• Resolved: 28/Oct/21 20:53

Description

I use InvokeScriptedProcessor

I'm trying to read a sensitive property from the process context

 

before restarting NiFi it was working fine

after restarting  NiFi - NiFi fails to startup with the error below

see https://issues.apache.org/jira/browse/NIFI-7012

 

 

2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalArgumentException: The property 'Password3' cannot reference Parameter 'password3' because Sensitive Parameters may only be referenced by Sensitive Properties.
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:423)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:387)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
at org.apache.nifi.NiFi.<init>(NiFi.java:159)
at org.apache.nifi.NiFi.<init>(NiFi.java:71)
at org.apache.nifi.NiFi.main(NiFi.java:303)
Caused by: java.lang.IllegalArgumentException: The property 'Password3' cannot reference Parameter 'password3' because Sensitive Parameters may only be referenced by Sensitive Properties.
at org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
at org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
at org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
at org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
at org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
at org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
... 43 common frames omitted

 

in flow.xml.gz

 

 

<parameterContext>
  <id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
  <name>otto_test</name>
  <description/>
  <parameter>
  <name>password</name>
  <description/>
  <sensitive>true</sensitive>
  <value>enc
{61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
</value>
  </parameter>
  <parameter>
  <name>password3</name>
  <description/>
  <sensitive>true</sensitive>
  <value>enc
{6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
</value>
  </parameter>
</parameterContext>

 

 

 

 

 

 

JIRA messed up and inserted lots of brackets {{ and }} all over the place, I tried to remove them but may have not got them all or to many removed

<processor>
  <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
  <name>InvokeScriptedProcessor2</name>
  <position x="-1144.0" y="3640.0"/>
  <styles/>
  <comment/>
  <class&amp;amp;gt;org.apache.nifi.processors.script.InvokeScriptedProcessor</class&amp;amp;gt;
  <bundle>
  <group>org.apache.nifi</group>
  <artifact>nifi-scripting-nar</artifact>
  <version>1.13.2</version>
  </bundle>
  <maxConcurrentTasks>1</maxConcurrentTasks>
  <schedulingPeriod>0 sec</schedulingPeriod>
  <penalizationPeriod>30 sec</penalizationPeriod>
  <yieldPeriod>1 sec</yieldPeriod>
  <bulletinLevel>WARN</bulletinLevel>
  <lossTolerant>false</lossTolerant>
  <scheduledState>RUNNING</scheduledState>
  <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
  <executionNode>ALL</executionNode>
  <runDurationNanos>0</runDurationNanos>
  <property>
  <name>Script Engine</name>
  <value>ECMAScript</value>
  </property>
  <property>
  <name>Script File</name>
  </property>
  <property>
  <name>Script Body</name>
  <value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
  //http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
  //http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
  //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
  //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
  //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
  //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
///////////////////////////////////////////////////////////
  // "imports" go here
  ////////////////////////////////////////////////////////////
  var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
  var IOUtils = Java.type("org.apache.commons.io.IOUtils");
  var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
  var Processor = Java.type("org.apache.nifi.processor.Processor");
  var Relationship = Java.type("org.apache.nifi.processor.Relationship");
  var StandardValidators = Java.type("org.apache.nifi.processor.util.StandardValidators");
  var Validator = Java.type("org.apache.nifi.components.Validator");
  
  var HashSet = Java.type("java.util.HashSet");
  var LinkedList = Java.type("java.util.LinkedList");
  var PropertyDescriptorBuilder = Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
  var log = null;
  var REL_SUCCESS = new Relationship.Builder().name("success").description('FlowFiles that were successfully processed are routed here').build();
  var REL_FAILURE = new Relationship.Builder().name("failure").description('FlowFiles that were not successfully processed are routed here').build();
  var strPassword2 = "Password2";
  var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
  .description("Password used to connect")
  .required(true)
  //.sensitive(false)
  //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
  .expressionLanguageSupported(true)
  .addValidator(Validator.VALID)
  .build();
  var strPassword3 = "Password3";
  var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
  .description("Password used to connect")
  .required(true)
  .sensitive(true)}}
  // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
  .expressionLanguageSupported(false)
  .addValidator(Validator.VALID)
  .build();

function executeScript(session, context, log, REL_SUCCESS, REL_FAILURE) {
  ////////////////////////////////////////////////////////////
  // your code goes here
  ////////////////////////////////////////////////////////////
  var pwd2 = context.getProperty(strPassword2).getValue();
  var pwd3 = context.getProperty(strPassword3).getValue();
  var pwd31 = context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
  log.error('pwd2:'+ pwd2);
  log.error('pwd3:'+ pwd3);
  var flowFile = session.get();
  if(flowFile != null) {}}if(flowFile != null) {
  // Create a new StreamCallback, passing in a function to define the interface method
  flowFile = session.write(flowFile,
  new StreamCallback(function(inputStream, outputStream){ 
var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8); 
outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8)); 
outputStream.write("\npwd2:".getBytes()); 
outputStream.write(pwd2.getBytes()); 
// outputStream.write("\npwd3:".getBytes()); 
// outputStream.write(pwd3.getBytes()); 
// outputStream.write("\n31:".getBytes()); 
// outputStream.write(pwd31.getBytes()); 
outputStream.write("\n".getBytes()); }
));
 
try{
 session.transfer(flowFile, REL_SUCCESS) 
}catch(e){
  log.error('Something went wrong', e) 
  session.transfer(flowFile, REL_FAILURE) 
}
}
function initialize(context){ log = context.logger; }

 function getRelationships()
{ 
  var r = new HashSet(); 
  r.add(REL_FAILURE); 
  r.add(REL_SUCCESS); 
  return r; 
}
function validate(context){ return null; }

function getPropertyDescriptor(name) {
  if(name.equals(strPassword2)){
     return prop_password2; 
  } else if(name.equals(strPassword3)){
     return prop_password3; 
  } else{{
     return null; 
  }
 }
function onPropertyModified(descriptor, oldValue, newValue) { return null; }

function getPropertyDescriptors(){
     var r = new LinkedList(); 
     r.add(prop_password2); 
     r.add(prop_password3); 
     return r; 
}
function getIdentifier(){ return null; }

function onTrigger(context, sessionFactory) {
  var session = sessionFactory.createSession();
  try{
    executeScript(session, context, log, REL_SUCCESS, REL_FAILURE); 
    session.commit(); 
  }catch (t) {
    log.error("{} failed to process due to {}; rolling back session", Java.to([this, t], "java.lang.Object[]"));
    session.rollback(true);
    throw t;
  }
}
processor = this;</value>
  </property>
  <property>
  <name>Module Directory</name>
  </property>
  <property>
  <name>Password2</name>
  <value>xyz</value>
  </property>
  <property>
  <name>Password3</name>
  <value>enc
{da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
</value>
  </property>
  </processor>
 

 

to recover I had to uncomment the code and remove the property:  Password3 using a text editor on the flow.xml

 

Many thanks to mburgess the cookbook author.

I'm new to NiFi so sorry if I've simply made a programming error.
It seems like I'm not the only one with the similar problem.

I try to createa flow that login to provenance repository in a NiFi with LDAP using the InvokeHttp in a later step. To login I need to fetch a JWT-token and then call the provenanc repo REST apis using the token. When fetching the token I want to hide tha password. Maybe there is a better way. I also would like to store the token in a secure way and not store it as attribute. I use the  DistributedMapCacheServer to cache the token until expiry but maybe there is a better way. 

https://host:port/nifi-api/access/token POST 

 This endpoint only supports form based authentication but if it supported basic authentcation too it would be simpler becasue then the InvokeHttp processor could access it and the password would be hidden by sensitive property. It doesn't help to actually hide the token in the flow. Howdo you store sensitve values at attribute in the flow? Maybe it is wrong thinking to do so.
It seems like many are struggling with oAuth2 token based authentication as well. client credentials grant and password credentials grant and so on. Havne't found any cookbook for that.

 

 

 I'm on Ubuntu and java 1.11 sap machine

In Dell Boomi they have https://ace.c9.io/ as script editor. It's almost a bug not to have a decent script editor for InvokeScriptedProcessor and the other script processors.  The tiny little NiFi script window is painful to wotk with and closes when you press enter to create a new line.

 

Issue Links

• is fixed by: NIFI-7012 Using sensitive parameter in sensitive property of InvokeScriptedProcessor causes Jetty shutdown on NiFi restart