Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8684

sensitive property not working for InvokeScriptedProcessor

    XMLWordPrintableJSON

    Details

      Description

      I use InvokeScriptedProcessor

      I'm trying to read a sensitive property from the process context

       

      before restarting NiFi it was working fine

      after restarting  NiFi - NiFi fails to startup with the error below

      see https://issues.apache.org/jira/browse/NIFI-7012

       

       

      2021-06-11 11:22:09,673 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
      org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalArgumentException: The property 'Password3' cannot reference Parameter 'password3' because Sensitive Parameters may only be referenced by Sensitive Properties.
      at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:306)
      at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1413)
      at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:89)
      at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:810)
      at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:539)
      at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:72)
      at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1068)
      at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
      at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:997)
      at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
      at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
      at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
      at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
      at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:911)
      at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
      at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
      at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:426)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
      at org.eclipse.jetty.server.Server.start(Server.java:423)
      at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
      at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
      at org.eclipse.jetty.server.Server.doStart(Server.java:387)
      at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
      at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1057)
      at org.apache.nifi.NiFi.<init>(NiFi.java:159)
      at org.apache.nifi.NiFi.<init>(NiFi.java:71)
      at org.apache.nifi.NiFi.main(NiFi.java:303)
      Caused by: java.lang.IllegalArgumentException: The property 'Password3' cannot reference Parameter 'password3' because Sensitive Parameters may only be referenced by Sensitive Properties.
      at org.apache.nifi.controller.AbstractComponentNode.verifyCanUpdateProperties(AbstractComponentNode.java:313)
      at org.apache.nifi.controller.AbstractComponentNode.setProperties(AbstractComponentNode.java:198)
      at org.apache.nifi.controller.ComponentNode.setProperties(ComponentNode.java:61)
      at org.apache.nifi.controller.StandardFlowSynchronizer.updateProcessor(StandardFlowSynchronizer.java:1249)
      at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessors(StandardFlowSynchronizer.java:1398)
      at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1317)
      at org.apache.nifi.controller.StandardFlowSynchronizer.addNestedProcessGroups(StandardFlowSynchronizer.java:1333)
      at org.apache.nifi.controller.StandardFlowSynchronizer.addProcessGroup(StandardFlowSynchronizer.java:1322)
      at org.apache.nifi.controller.StandardFlowSynchronizer.updateFlow(StandardFlowSynchronizer.java:432)
      at org.apache.nifi.controller.StandardFlowSynchronizer.sync(StandardFlowSynchronizer.java:283)
      ... 43 common frames omitted

       

      in flow.xml.gz

       

       

      <parameterContext>
        <id>f8098456-0179-1000-40b0-ba2fcee07a72</id>
        <name>otto_test</name>
        <description/>
        <parameter>
        <name>password</name>
        <description/>
        <sensitive>true</sensitive>
        <value>enc
      {61d12bbcfe272e47e344cb7d3ee04ff68d2ca5b9a2ecfae2bf2f42e1eb2e6230}
      </value>
        </parameter>
        <parameter>
        <name>password3</name>
        <description/>
        <sensitive>true</sensitive>
        <value>enc
      {6ee4436cc391ac9a0cb567a3f7a7c26155b4b353152b52325f67eafa6b106c0b}
      </value>
        </parameter>
      </parameterContext>
      

       

       

       

       

       

       

      JIRA messed up and inserted lots of brackets {{ and }} all over the place, I tried to remove them but may have not got them all or to many removed
      <processor>
        <id>f7bd4a82-0179-1000-484a-eb7fbafde530</id>
        <name>InvokeScriptedProcessor2</name>
        <position x="-1144.0" y="3640.0"/>
        <styles/>
        <comment/>
        <class&amp;amp;gt;org.apache.nifi.processors.script.InvokeScriptedProcessor</class&amp;amp;gt;
        <bundle>
        <group>org.apache.nifi</group>
        <artifact>nifi-scripting-nar</artifact>
        <version>1.13.2</version>
        </bundle>
        <maxConcurrentTasks>1</maxConcurrentTasks>
        <schedulingPeriod>0 sec</schedulingPeriod>
        <penalizationPeriod>30 sec</penalizationPeriod>
        <yieldPeriod>1 sec</yieldPeriod>
        <bulletinLevel>WARN</bulletinLevel>
        <lossTolerant>false</lossTolerant>
        <scheduledState>RUNNING</scheduledState>
        <schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
        <executionNode>ALL</executionNode>
        <runDurationNanos>0</runDurationNanos>
        <property>
        <name>Script Engine</name>
        <value>ECMAScript</value>
        </property>
        <property>
        <name>Script File</name>
        </property>
        <property>
        <name>Script Body</name>
        <value>//http://funnifi.blogspot.com/2018/02/invokescriptedprocessor-template.html
        //http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
        //http://funnifi.blogspot.com/2016/02/invokescriptedprocessor-hello-world.html
        //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/package-summary.html
        //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/index.html?org/apache/nifi/processor/ProcessSession.html
        //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/components/PropertyValue.html
        //https://www.javadoc.io/static/org.apache.nifi/nifi-api/1.13.2/org/apache/nifi/context/package-frame.html
      ///////////////////////////////////////////////////////////
        // "imports" go here
        ////////////////////////////////////////////////////////////
        var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
        var IOUtils = Java.type("org.apache.commons.io.IOUtils");
        var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
        var Processor = Java.type("org.apache.nifi.processor.Processor");
        var Relationship = Java.type("org.apache.nifi.processor.Relationship");
        var StandardValidators = Java.type("org.apache.nifi.processor.util.StandardValidators");
        var Validator = Java.type("org.apache.nifi.components.Validator");
        
        var HashSet = Java.type("java.util.HashSet");
        var LinkedList = Java.type("java.util.LinkedList");
        var PropertyDescriptorBuilder = Java.type("org.apache.nifi.components.PropertyDescriptor.Builder");
        var log = null;
        var REL_SUCCESS = new Relationship.Builder().name("success").description('FlowFiles that were successfully processed are routed here').build();
        var REL_FAILURE = new Relationship.Builder().name("failure").description('FlowFiles that were not successfully processed are routed here').build();
        var strPassword2 = "Password2";
        var prop_password2 = new PropertyDescriptorBuilder().name(strPassword2)
        .description("Password used to connect")
        .required(true)
        //.sensitive(false)
        //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
        .expressionLanguageSupported(true)
        .addValidator(Validator.VALID)
        .build();
        var strPassword3 = "Password3";
        var prop_password3 = new PropertyDescriptorBuilder().name(strPassword3)
        .description("Password used to connect")
        .required(true)
        .sensitive(true)}}
        // //.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
        .expressionLanguageSupported(false)
        .addValidator(Validator.VALID)
        .build();
      
      function executeScript(session, context, log, REL_SUCCESS, REL_FAILURE) {
        ////////////////////////////////////////////////////////////
        // your code goes here
        ////////////////////////////////////////////////////////////
        var pwd2 = context.getProperty(strPassword2).getValue();
        var pwd3 = context.getProperty(strPassword3).getValue();
        var pwd31 = context.getProperty(strPassword3).evaluateAttributeExpressions().getValue();
        log.error('pwd2:'+ pwd2);
        log.error('pwd3:'+ pwd3);
        var flowFile = session.get();
        if(flowFile != null) {}}if(flowFile != null) {
        // Create a new StreamCallback, passing in a function to define the interface method
        flowFile = session.write(flowFile,
        new StreamCallback(function(inputStream, outputStream){ 
      var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8); 
      outputStream.write(text.split("").reverse().join("").getBytes(StandardCharsets.UTF_8)); 
      outputStream.write("\npwd2:".getBytes()); 
      outputStream.write(pwd2.getBytes()); 
      // outputStream.write("\npwd3:".getBytes()); 
      // outputStream.write(pwd3.getBytes()); 
      // outputStream.write("\n31:".getBytes()); 
      // outputStream.write(pwd31.getBytes()); 
      outputStream.write("\n".getBytes()); }
      ));
       
      try{
       session.transfer(flowFile, REL_SUCCESS) 
      }catch(e){
        log.error('Something went wrong', e) 
        session.transfer(flowFile, REL_FAILURE) 
      }
      }
      function initialize(context){ log = context.logger; }
      
       function getRelationships()
      { 
        var r = new HashSet(); 
        r.add(REL_FAILURE); 
        r.add(REL_SUCCESS); 
        return r; 
      }
      function validate(context){ return null; }
      
      function getPropertyDescriptor(name) {
        if(name.equals(strPassword2)){
           return prop_password2; 
        } else if(name.equals(strPassword3)){
           return prop_password3; 
        } else{{
           return null; 
        }
       }
      function onPropertyModified(descriptor, oldValue, newValue) { return null; }
      
      function getPropertyDescriptors(){
           var r = new LinkedList(); 
           r.add(prop_password2); 
           r.add(prop_password3); 
           return r; 
      }
      function getIdentifier(){ return null; }
      
      function onTrigger(context, sessionFactory) {
        var session = sessionFactory.createSession();
        try{
          executeScript(session, context, log, REL_SUCCESS, REL_FAILURE); 
          session.commit(); 
        }catch (t) {
          log.error("{} failed to process due to {}; rolling back session", Java.to([this, t], "java.lang.Object[]"));
          session.rollback(true);
          throw t;
        }
      }
      processor = this;</value>
        </property>
        <property>
        <name>Module Directory</name>
        </property>
        <property>
        <name>Password2</name>
        <value>xyz</value>
        </property>
        <property>
        <name>Password3</name>
        <value>enc
      {da09776f7e7ffda657ddff989489c93cae2821b8a1fcded6d2d4d5c1964187da}
      </value>
        </property>
        </processor>
       
      

       

      to recover I had to uncomment the code and remove the property:  Password3 using a text editor on the flow.xml

       

      Many thanks to mburgess the cookbook author.

      I'm new to NiFi so sorry if I've simply made a programming error.
      It seems like I'm not the only one with the similar problem.

      I try to createa flow that login to provenance repository in a NiFi with LDAP using the InvokeHttp in a later step. To login I need to fetch a JWT-token and then call the provenanc repo REST apis using the token. When fetching the token I want to hide tha password. Maybe there is a better way. I also would like to store the token in a secure way and not store it as attribute. I use the  DistributedMapCacheServer to cache the token until expiry but maybe there is a better way. 

      https://host:port/nifi-api/access/token POST 

       This endpoint only supports form based authentication but if it supported basic authentcation too it would be simpler becasue then the InvokeHttp processor could access it and the password would be hidden by sensitive property. It doesn't help to actually hide the token in the flow. Howdo you store sensitve values at attribute in the flow? Maybe it is wrong thinking to do so.
      It seems like many are struggling with oAuth2 token based authentication as well. client credentials grant and password credentials grant and so on. Havne't found any cookbook for that.

       

       

       I'm on Ubuntu and java 1.11 sap machine

      In Dell Boomi they have https://ace.c9.io/ as script editor. It's almost a bug not to have a decent script editor for InvokeScriptedProcessor and the other script processors.  The tiny little NiFi script window is painful to wotk with and closes when you press enter to create a new line.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              tomten1970 Jul Tomten
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: